oglyad 0.1.4.0 → 0.1.6.0
raw patch · 31 files changed
+1474/−1140 lines, 31 filesdep ~QuickCheckdep ~aesondep ~async
Dependency ranges changed: QuickCheck, aeson, async, base, bytestring, conduit, containers, deepseq, filepath, hspec, http-client, megaparsec, text, typed-process, vector, xml-conduit
Files
- CHANGELOG.md +55/−0
- README.md +47/−12
- app/Main.hs +107/−67
- oglyad.cabal +20/−19
- src/Oglyad/App.hs +57/−31
- src/Oglyad/Check.hs +6/−6
- src/Oglyad/Check/BuildLag.hs +3/−6
- src/Oglyad/Check/Drift.hs +11/−16
- src/Oglyad/Check/DuplicateName.hs +3/−5
- src/Oglyad/Check/FedoraOverlap.hs +4/−5
- src/Oglyad/Check/NoAutoupdate.hs +0/−1
- src/Oglyad/Check/Stale.hs +2/−4
- src/Oglyad/Check/Submodules.hs +25/−0
- src/Oglyad/Check/UnparsedRhai.hs +5/−9
- src/Oglyad/Check/UpstreamError.hs +0/−2
- src/Oglyad/Collect.hs +126/−101
- src/Oglyad/Ignore.hs +11/−32
- src/Oglyad/Process.hs +22/−27
- src/Oglyad/Regex.hs +99/−123
- src/Oglyad/Release.hs +234/−72
- src/Oglyad/Report.hs +14/−31
- src/Oglyad/Source/Anda.hs +95/−36
- src/Oglyad/Source/Cache.hs +22/−22
- src/Oglyad/Source/Monorepo.hs +81/−95
- src/Oglyad/Source/Repodata.hs +35/−71
- src/Oglyad/Source/Rhai.hs +36/−58
- src/Oglyad/Source/RhaiEval.hs +74/−101
- src/Oglyad/Source/Upstream.hs +32/−29
- src/Oglyad/Types.hs +60/−71
- src/Oglyad/Version.hs +28/−43
- test/Spec.hs +160/−45
CHANGELOG.md view
@@ -1,5 +1,60 @@ # changelog +## 0.1.6.0++- new `submodules` check (`-u`): update.rhai scripts that call+ `rpm.submodules()` run under anda, and a submodule upstream added without a+ `%global submodule_*`, or one the spec still has after upstream dropped it,+ is a warning. needs an anda with `rpm.submodules()`. the first run after+ upgrading reruns every script under anda, since its cache now keeps the+ submodule changes too+- scripts run under anda are read from `anda update --report` instead of+ anda's log, so a failing helper shows up as `gh(): status code 404 (line 3)`+ instead of the first line of the traceback. this needs an anda with+ `--report`. an older one is a `collect` warning and those scripts are+ skipped+- builds with GHC 9.6 and the libraries EPEL 10 ships, so the el10 package+ no longer has to strip version bounds. CI builds against both Fedora and+ EPEL 10++## 0.1.5.0++- redo comments+- checks Terra EL too: `-d`/`--distro terra-el` for terrapkg/packages-el,+ picked automatically when the remote's url names packages-el, or+ `-r el10` on a terrapkg/packages checkout until the split is done. specs+ see EL's macros (`%rhel`, `%epel`, `%el10`, `.el10`), builds are read+ from `terrael10`, and `--base` compares against AlmaLinux and EPEL.+ terra-el checks every `elN` branch by default, terra only checks EL when+ asked+- checks terrapkg/tuatara too: `-d`/`--distro tuatara`, or picked+ automatically from a checkout with a `tuatara/` directory. its one+ release is `tumbleweed`, specs see openSUSE's macros instead of Fedora's,+ and builds are read from `tuatara-tumbleweed` on repos.fyralabs.com+- `--fedora` and `--fedora-repo` are now `--base` and `--base-repo`, since+ the overlap check uses whichever distro the monorepo builds on. the old+ names still work+- cache files are named per distro and release instead of per dist tag, so+ the first run after upgrading reevaluates every spec+- releases are checked in parallel instead of one after another. a cold run+ over f43, f44, f45 and rawhide went from about 4 minutes to 1.5. progress+ messages say which release they're about when checking more than one+- update.rhai scripts without an `updbranch` label only run under anda on+ rawhide, like Terra's update workflows, which update rawhide and copy the+ result to the release branches. `unparsed-rhai` only reports them on+ rawhide too+- upstream lookups and anda runs happen at the same time+- an upstream that several releases ask about is looked up once, even when+ the lookup fails+- rpmspec and anda share one pool of `--jobs` slots across all releases+- the missing `GITHUB_TOKEN` warning is printed once per run+- `no-autoupdate` doesn't run on Ultramarine, which updates most packages by+ hand on purpose+- rpmspec refusing a spec's `BuildArch` isn't reported when the package's+ anda.hcl `arches` leave out this machine's arch anyway, like a+ `BuildArch: aarch64` package checked from x86_64. other rpmspec failures+ are still reported, including for noarch packages pinned to one builder+ ## 0.1.4.0 - check Bodhi multiple times and cache response
README.md view
@@ -8,11 +8,14 @@ oglyad -m ../packages # warnings and errors, every maintained release oglyad -m ../packages -v # plus info findings oglyad -m ../packages -u # plus upstream version drift-oglyad -m ../packages --fedora # plus overlap with Fedora+oglyad -m ../packages --base # plus overlap with Fedora (or EL, or openSUSE) oglyad -m ../packages -r 44 -r rawhide # just these releases oglyad -m ../packages -r rawhide --worktree # your local edits, against rawhide+oglyad -m ../packages-el # Terra EL, from terrapkg/packages-el+oglyad -m ../packages -r el10 # or from packages' el10 branch, until the split is done oglyad -m ../packages --only drift --format json oglyad -m ../um-packages # Ultramarine-Linux/packages works too+oglyad -m ../tuatara # so does terrapkg/tuatara, on openSUSE ``` from a checkout, `cabal run oglyad -- <args>` and `cabal test`. it needs@@ -25,7 +28,7 @@ [Bodhi](https://bodhi.fedoraproject.org/releases/) hasn't archived (the current ones, a branched one before its release, and rawhide), so branching and EOL need no changes here. `-r`/`--release` picks releases instead, as-`43`, `f43` or `rawhide`, and can be repeated.+`43`, `f43`, `rawhide` or `el10`, and can be repeated. each release is checked against its own Terra repo (`terra43`, `terrarawhide`) and its own branch of terrapkg/packages (`origin/f43`,@@ -41,16 +44,39 @@ ## distros and repos oglyad checks any anda monorepo it knows the layout of. `-d`/`--distro`-picks one, and by default it's whichever one's package directory the-checkout has:+picks one, and by default it's whichever one the checkout looks like (it+has `anda/`, `ultramarine/` or `tuatara/`). terrapkg/packages and+packages-el both have `anda/`, so those two are told apart by the repo name+in the remote's url (`--remote`, `origin` by default). a fork with another+name needs `--distro`: -| distro | monorepo | packages in | branches | repos | dist |-| ------------- | -------------------------- | -------------- | ------------------- | --------------------------- | --------------------- |-| `terra` | terrapkg/packages | `anda/` | `f43`, `frawhide` | `terra43`, `terra43-extras` | `.fc43`, `.fcrawhide` |-| `ultramarine` | Ultramarine-Linux/packages | `ultramarine/` | `um43`, `umrawhide` | `um43` | `.um43`, `.umrawhide` |+| distro | base | monorepo | packages in | branches | repos | dist |+| ------------- | -------- | -------------------------- | -------------- | ------------------- | --------------------------- | --------------------- |+| `terra` | Fedora | terrapkg/packages | `anda/` | `f43`, `frawhide` | `terra43`, `terra43-extras` | `.fc43`, `.fcrawhide` |+| `terra` | EL | terrapkg/packages | `anda/` | `el10` | `terrael10` | `.el10` |+| `terra-el` | EL | terrapkg/packages-el | `anda/` | `el10` | `terrael10` | `.el10` |+| `ultramarine` | Fedora | Ultramarine-Linux/packages | `ultramarine/` | `um43`, `umrawhide` | `um43` | `.um43`, `.umrawhide` |+| `tuatara` | openSUSE | terrapkg/tuatara | the top level | `tumbleweed` | `tuatara-tumbleweed` | none | +Tuatara only builds for Tumbleweed, so that's the one release it has+(`-r tumbleweed`, or `tw`), and Bodhi isn't asked. its specs are evaluated+with `%suse_version` 1699 and `%is_opensuse` 1, and with `%fedora` and+`%dist` undefined, like on a Tumbleweed builder. subrepo labels don't get+their own repo there, since every build lands in `tuatara-tumbleweed`.++Terra EL is moving from terrapkg/packages' `el10` branch to+terrapkg/packages-el. a packages-el checkout checks every `elN` branch the+remote has by default (Bodhi lists EPEL releases Terra doesn't build, so+it isn't asked). a packages checkout only checks EL with `-r el10`. specs are evaluated with `%rhel` and `%epel`+10, `%el10` 1, `%dist` `.el10` and `%fedora` undefined, like on Terra's EL+builder. every EL build lands in `terrael10`, so subrepo labels don't get+their own repo, and `--base` checks overlap with AlmaLinux's BaseOS,+AppStream, CRB and extras repos plus EPEL.+ builds are looked up through the same metalink dnf uses,-`https://tetsudou.fyralabs.com/metalink?repo={repo}&arch=x86_64`. oglyad+`https://tetsudou.fyralabs.com/metalink?repo={repo}&arch=x86_64`, except+for Tuatara, which tetsudou has no mirrors for, so it's read straight from+`https://repos.fyralabs.com/{repo}`. oglyad tries its mirrors by preference and takes the first repomd.xml whose sha256 the metalink lists, so a mirror that hasn't synced yet is skipped. @@ -141,7 +167,8 @@ | `duplicate-name` | warn | two directories build the same package name into the same repo, so they overwrite each other | | `stale` | info, warn with `-u` | last build is older than `--stale-days`, warn if upstream also moved on | | `upstream-error` | warn | the upstream from update.rhai 404s or fails, so `anda update` is failing too |-| `fedora-overlap` | warn | Fedora ships the same source package (`--fedora`). subrepos are exempt |+| `submodules` | warn | upstream added a submodule the spec has no `%global submodule_*` for, or dropped one it still has (`-u`). comes from anda running a script that calls `rpm.submodules()`, see below |+| `fedora-overlap` | warn | the base distro ships the same source package (`--base`, or `--fedora`): Fedora's release and updates repos, AlmaLinux and EPEL for `el10`, or Tumbleweed's oss and update repos for Tuatara. subrepos are exempt | | `no-autoupdate` | info | no update.rhai, or one with only comments in it | | `unparsed-rhai` | info, warn when anda can't run it either | update.rhai looks a version up in a way oglyad can't follow, like custom scraping, and anda didn't answer for it (no `-u`, or anda isn't installed). scripts that only bump Release, track a commit, or copy Version from another spec aren't listed | | `collect` | error | rpmspec failed, a repo couldn't be fetched, or an upstream rate-limited us (warn) |@@ -157,16 +184,24 @@ instead. with `-u` and `anda` on the PATH, oglyad copies the release's tree to a throwaway directory, runs `anda update --labels branch=<f43> <project>` for each of those, and reads the Version rpmspec sees in the rewritten spec.-a script that throws shows up as `upstream-error`. answers are cached like+whether a script failed comes from the `--report` json anda writes, so this+needs an anda new enough to have `anda update --report`. (TODO: add version)+like Terra's update workflows, only projects with an `updbranch` label run on+release branches. the rest only run on rawhide. a script that throws shows up as `upstream-error`. answers are cached like upstream ones. +scripts that call `rpm.submodules()` always go to anda too, even when oglyad+can follow their version, since only anda knows what the submodules are. the+submodules anda reports as added or dropped become `submodules` warnings. a+submodule in the script's `ignore` list isn't reported.+ ## how it works ``` ┌──────────── IO shell (Collect) ──┐ ┌──── pure core ────┐ ┌─ IO ─┐ monorepo: rpmspec, git ──┐ Terra repodata + subrepos├─► Snapshot ───► [Check] ─► [Finding] ─► Report- Fedora repodata ─────────┤+ Fedora/SUSE repodata ────┤ upstream APIs ───────────┘ ```
app/Main.hs view
@@ -6,16 +6,18 @@ import Control.Concurrent (threadDelay) import Control.Exception (SomeException, try)-import Control.Monad (filterM, forM, unless, when)+import Control.Monad (filterM, unless, when) import Data.ByteString qualified as BS import Data.ByteString.Lazy qualified as LBS+import Data.Containers.ListUtils (nubOrd) import Data.List (nub)+import Data.Maybe (fromMaybe, isNothing) import Data.Text (Text) import Data.Text qualified as T import Data.Time (UTCTime, getCurrentTime) import GHC.Conc (getNumProcessors) import GHC.IO.Encoding (setFileSystemEncoding, setForeignEncoding, setLocaleEncoding, utf8)-import Oglyad.App (HttpError (..), httpGetE, oglyadVersion, runApp)+import Oglyad.App (HttpError (..), forConcurrentlyN, httpGetE, inRelease, note, oglyadVersion, runApp) import Oglyad.Check (checkNames, parseCheckName, runChecks) import Oglyad.Collect (buildSnapshot) import Oglyad.Ignore (Applied (..), IgnoreFile (..), applyIgnores, parseIgnoreFile)@@ -32,16 +34,21 @@ import System.FilePath ((</>)) import System.IO (hPutStrLn, hSetEncoding, stderr, stdout) --- | exit codes, so cron jobs and CI can tell "found problems" apart from--- "couldn't run" exitFindings, exitUsage :: ExitCode exitFindings = ExitFailure 1 exitUsage = ExitFailure 2 +data ConfigInputs = ConfigInputs+ { ciGitHubToken :: Maybe Text,+ ciCores :: Int,+ ciRunStart :: UTCTime,+ ciSpecRef :: Maybe Text,+ ciDistro :: Distro,+ ciRelease :: Release+ }+ data Opts = Opts- { -- | token, cores, run start, spec ref, distro, release- optConfig :: Maybe Text -> Int -> UTCTime -> Maybe Text -> Distro -> Release -> Config,- -- | Nothing means tell from the checkout+ { optConfig :: ConfigInputs -> Config, optDistro :: Maybe Distro, optReleases :: [ReleaseArg], optRemote :: Text,@@ -50,7 +57,6 @@ optOnly :: [Text], optFormat :: Format, optMinSev :: Severity,- -- | Nothing means never fail on findings optFailOn :: Maybe Severity, optIgnoreFile :: Maybe FilePath, optNoIgnores :: Bool@@ -59,11 +65,11 @@ optsP :: Parser Opts optsP = do optConfig <- configP- optDistro <- optional (option (eitherReader lookupDistro) (long "distro" <> short 'd' <> metavar "NAME" <> help ("which monorepo -m is: " <> distroNames <> ". default: whichever one's package directory (anda/, ultramarine/) the checkout has")))- optReleases <- many (option (eitherReader parseReleaseArg) (long "release" <> short 'r' <> metavar "RELEASE" <> help "release to check: 43, f43 or rawhide. repeatable. default: every release Fedora's Bodhi hasn't archived"))- optRemote <- strOption (long "remote" <> metavar "NAME" <> value "origin" <> showDefault <> help "git remote whose release branches (f43, frawhide, um43, ...) hold each release's specs. nothing is fetched, run git fetch yourself")+ optDistro <- optional (option (eitherReader lookupDistro) (long "distro" <> short 'd' <> metavar "NAME" <> help ("which monorepo -m is: " <> distroNames <> ". default: whichever one the checkout looks like (it has anda/, ultramarine/ or tuatara/, and the remote's repo name tells packages and packages-el apart)")))+ optReleases <- many (option (eitherReader parseReleaseArg) (long "release" <> short 'r' <> metavar "RELEASE" <> help "release to check: 43, f43 or rawhide, el10 for Terra EL, or tumbleweed for openSUSE-based distros. repeatable. default: every release Fedora's Bodhi hasn't archived, every elN branch for terra-el, or tumbleweed"))+ optRemote <- strOption (long "remote" <> metavar "NAME" <> value "origin" <> showDefault <> help "git remote whose release branches (f43, frawhide, el10, um43, tumbleweed, ...) hold each release's specs. nothing is fetched, run git fetch yourself") optWorktree <- switch (long "worktree" <> help "read specs from the checkout as it is, local edits included, instead of the release branch. needs exactly one --release")- optRepo <- optional (strOption (long "repo" <> metavar "URL" <> help ("metalink or base url of the repo to check builds against. {repo} becomes the repo's name (terra43, terra43-extras) and {release} the release's (43, rawhide). a url with neither names one release's repo, so it needs exactly one --release. default: " <> T.unpack defaultRepoUrl)))+ optRepo <- optional (strOption (long "repo" <> metavar "URL" <> help ("metalink or base url of the repo to check builds against. {repo} becomes the repo's name (terra43, terra43-extras) and {release} the release's (43, rawhide). a url with neither names one release's repo, so it needs exactly one --release. default: " <> T.unpack defaultRepoUrl <> ", or https://repos.fyralabs.com/{repo} for tuatara"))) optOnly <- many (option (eitherReader parseCheckName) (long "only" <> metavar "CHECK" <> help ("run only this check, repeatable. one of: " <> names))) optFormat <- option (eitherReader format) (long "format" <> short 'f' <> metavar "FORMAT" <> help "text (default), json, or markdown")@@ -91,48 +97,69 @@ "never" -> Right Nothing other -> Left ("unknown level " <> show other <> ", expected error, warn, info or never") -configP :: Parser (Maybe Text -> Int -> UTCTime -> Maybe Text -> Distro -> Release -> Config)+configP :: Parser (ConfigInputs -> Config) configP = do- cfgMonorepo <- strOption (long "monorepo" <> short 'm' <> metavar "DIR" <> value "." <> showDefault <> help "checkout of terrapkg/packages, or Ultramarine-Linux/packages with --distro ultramarine")- fedora <- switch (long "fedora" <> help "also check for overlap with Fedora's release and updates repos (downloads ~60MB per release)")- extraFedora <- many (strOption (long "fedora-repo" <> metavar "URL" <> help "extra repo base url to check for overlap, repeatable"))+ cfgMonorepo <- strOption (long "monorepo" <> short 'm' <> metavar "DIR" <> value "." <> showDefault <> help "checkout of terrapkg/packages, terrapkg/packages-el, Ultramarine-Linux/packages or terrapkg/tuatara")+ base <- switch (long "base" <> long "fedora" <> help "also check for overlap with the base distro's repos: Fedora's release and updates (~60MB per release), AlmaLinux and EPEL for el releases, or openSUSE Tumbleweed's oss and update")+ extraBase <- many (strOption (long "base-repo" <> long "fedora-repo" <> metavar "URL" <> help "extra repo base url to check for overlap, repeatable")) staleDays <- option auto (long "stale-days" <> metavar "N" <> value (180 :: Int) <> showDefault <> help "builds older than this count as stale") cfgUpstream <- switch (long "upstream" <> short 'u' <> help "query upstream APIs for version drift") cacheHours <- option auto (long "cache-hours" <> metavar "N" <> value (6 :: Int) <> showDefault <> help "reuse upstream answers younger than this, 0 to always refetch") jobs <- optional (option auto (long "jobs" <> short 'j' <> metavar "N" <> help "parallel jobs (default: number of cores)"))- pure $ \cfgGitHubToken cores cfgRunStart cfgSpecRef distro rel ->+ pure $ \ConfigInputs {ciDistro = distro, ciRelease = rel, ..} -> Config- { cfgRoot = distroRoot distro,+ { cfgGitHubToken = ciGitHubToken,+ cfgRunStart = ciRunStart,+ cfgSpecRef = ciSpecRef,+ cfgRoot = distroRoot distro, cfgRepoName = repoName distro rel,- cfgRepoUrl = repoUrlFor defaultRepoUrl rel,- cfgFedoraRepos = (if fedora then fedoraReposFor rel else []) <> extraFedora,- cfgRelease = relVersion rel,- cfgDist = releaseDist distro rel,+ cfgRepoUrl = repoUrlFor (distroRepoUrl distro) rel,+ cfgSubrepos = releaseSubrepos distro rel,+ cfgBaseRepos = (if base then baseReposFor rel else []) <> extraBase,+ cfgBaseName = baseRepoOwners (relBase rel),+ cfgMacros = releaseMacros distro rel,+ cfgCacheTag = releaseCacheTag distro rel, cfgBranch = releaseBranch distro rel,+ cfgRolling = isRolling rel, cfgStaleAfter = fromIntegral (staleDays * 86400), cfgCacheFor = fromIntegral (cacheHours * 3600),- cfgConcurrency = maybe cores id jobs,+ cfgConcurrency = fromMaybe ciCores jobs, .. } --- | turn --release arguments into releases, asking Bodhi for the--- maintained ones when none were given, and for rawhide's number when--- rawhide was asked for-resolveReleases :: (Release -> Config) -> [ReleaseArg] -> IO [Release]-resolveReleases mk args- | null args = bodhi >>= either (\e -> runError ("can't ask Bodhi which releases are maintained: " <> e <> ". pass --release to pick them")) pure+-- | EL releases come from the remote's elN branches, not Bodhi, which lists+-- EPEL releases Terra doesn't build+resolveReleases :: FilePath -> Text -> Distro -> (Release -> Config) -> [ReleaseArg] -> IO [Release]+resolveReleases mono remote distro mk args+ | null args = case distroBases distro of+ OpenSUSE : _ -> pure [tumbleweed]+ EL : _ ->+ elReleasesFrom <$> remoteBranches mono remote >>= \case+ [] -> runError ("the " <> remote <> " remote has no el10-style branches, so there's no EL release to check. git fetch, or pass --release")+ rels -> pure rels+ _ -> bodhi >>= either (\e -> runError ("can't ask Bodhi which releases are maintained: " <> e <> ". pass --release to pick them")) pure | otherwise = nub <$> mapM resolve args where- resolve = \case- NumberedArg n -> pure (Release (T.pack (show n)) n)+ resolve arg+ | releaseArgBase arg `notElem` distroBases distro =+ runError (distroName distro <> " is built on " <> T.intercalate " and " (map baseName (distroBases distro)) <> ", so it has no " <> baseName (releaseArgBase arg) <> " releases")+ resolve arg = case arg of+ TumbleweedArg -> pure tumbleweed+ ElArg n -> pure (el n)+ NumberedArg n -> pure (Release Fedora (T.pack (show n)) n) RawhideArg -> bodhi >>= \case- Right rels | r : _ <- filter isRawhide rels -> pure r+ Right rels | r : _ <- filter isRolling rels -> pure r Right _ -> runError "Bodhi didn't list a rawhide release" Left e -> runError ("can't ask Bodhi for rawhide's version: " <> e <> ". pass numbered releases instead")- bodhi = askBodhiPatiently (mk (Release "rawhide" 0))+ bodhi = askBodhiPatiently (mk anyRelease) runError msg = hPutStrLn stderr ("oglyad: " <> T.unpack msg) >> exitWith exitUsage +remoteBranches :: FilePath -> Text -> IO [Text]+remoteBranches mono remote =+ either (const []) T.lines+ <$> run "git" ["-C", mono, "for-each-ref", "--format=%(refname:lstrip=3)", "refs/remotes/" <> T.unpack remote <> "/"]+ bodhiAttempts :: Int bodhiAttempts = 4 @@ -159,7 +186,7 @@ loadJson bodhiCacheFile >>= \case Just (fetched :: UTCTime, rels) -> do warn ("Bodhi didn't answer (" <> e <> "), using its release list from " <> T.pack (show fetched))- pure (Right [Release name v | (name, v) <- rels])+ pure (Right [Release Fedora name v | (name, v) <- rels]) Nothing -> pure (Left e) warn msg = hPutStrLn stderr ("oglyad: " <> T.unpack msg) @@ -171,9 +198,8 @@ Left (RateLimited e) -> Left e Right b -> parseBodhiReleases b --- | GITHUB_TOKEN, or whatever the gh cli is logged in with. oglyad itself--- only sends it to api.github.com. the one child process that gets it is--- anda (see Oglyad.Source.Anda), since its GitHub helpers need it+-- | oglyad only sends this to api.github.com. the one child process that gets+-- it is anda, since its GitHub helpers need it githubToken :: IO (Maybe Text) githubToken = lookupEnv "GITHUB_TOKEN" >>= \case@@ -182,7 +208,6 @@ where nonEmpty t = if T.null t then Nothing else Just t --- | like execParser, but bad arguments exit with 'exitUsage' instead of 1 parseOpts :: IO Opts parseOpts = do args <- getArgs@@ -193,38 +218,48 @@ (fullDesc <> progDesc "repository metadata and health checker for Terra and other anda monorepos") case execParserPure defaultPrefs pinfo args of Failure f -> case renderFailure f prog of- -- --help and --version are output, not errors, so they go to stdout- -- (help2man and friends read them from there)+ -- help2man and friends read --help and --version from stdout (msg, ExitSuccess) -> putStrLn msg >> exitWith ExitSuccess (msg, _) -> hPutStrLn stderr msg >> exitWith exitUsage other -> handleParseResult other --- | the distro asked for, or the one whose package directory the checkout--- has. a checkout with none, or with more than one, has to say-pickDistro :: FilePath -> Maybe Distro -> IO Distro-pickDistro mono asked = do- found <- filterM (doesDirectoryExist . (mono </>) . distroRoot) distros+pickDistro :: FilePath -> Text -> Maybe Distro -> IO Distro+pickDistro mono remote asked = do+ found <- filterM (doesDirectoryExist . (mono </>) . distroMarker) distros case asked of Just d | d `elem` found -> pure d- | otherwise -> usageError (mono <> " doesn't look like a " <> T.unpack (distroName d) <> " checkout (no " <> distroRoot d <> "/ directory). point -m at one")+ | otherwise -> usageError (mono <> " doesn't look like a " <> T.unpack (distroName d) <> " checkout (no " <> distroMarker d <> "/ directory). point -m at one") Nothing -> case found of [d] -> pure d- [] -> usageError (mono <> " doesn't look like an anda monorepo (no " <> roots <> " directory). point -m at a checkout of terrapkg/packages or another, see --distro")- _ -> usageError (mono <> " has more than one package directory (" <> T.unpack (T.intercalate ", " (map distroName found)) <> "), so pick one with --distro")+ [] -> usageError (mono <> " doesn't look like an anda monorepo (no " <> markers <> " directory). point -m at a checkout of terrapkg/packages or another, see --distro")+ _ -> do+ upstream <- either (const Nothing) remoteRepoName <$> run "git" ["-C", mono, "remote", "get-url", T.unpack remote]+ case [d | d <- found, Just (distroUpstream d) == upstream] of+ [d] -> pure d+ _ -> usageError (mono <> " looks like more than one distro (" <> T.unpack (T.intercalate ", " (map distroName found)) <> ") and its " <> T.unpack remote <> " remote doesn't say which, so pick one with --distro") where- roots = T.unpack (T.intercalate "/, " (map (T.pack . distroRoot) distros)) <> "/"+ markers = T.unpack (T.intercalate "/, " (nubOrd (map (T.pack . distroMarker) distros))) <> "/" usageError :: String -> IO a usageError msg = hPutStrLn stderr ("oglyad: " <> msg) >> exitWith exitUsage -main :: IO ()-main = do- -- cron and CI often run under LANG=C. specs, anda.hcl and rpmspec errors- -- contain non-ascii text, which would crash text IO in a C locale+-- | cron and CI often run under LANG=C, where the non-ascii text in specs,+-- anda.hcl and rpmspec errors would crash text IO+useUtf8Everywhere :: IO ()+useUtf8Everywhere = do mapM_ ($ utf8) [setLocaleEncoding, setFileSystemEncoding, setForeignEncoding] mapM_ (`hSetEncoding` utf8) [stdout, stderr] +anyRelease :: Release+anyRelease = Release Fedora "rawhide" 0++namesOneRepo :: Text -> Bool+namesOneRepo url = not (any (`T.isInfixOf` url) ["{repo}", "{release}"])++main :: IO ()+main = do+ useUtf8Everywhere Opts {..} <- parseOpts ignores <- case optIgnoreFile of Nothing -> pure (IgnoreFile [])@@ -235,38 +270,43 @@ Right bytes -> either (\e -> usageError (path <> ": " <> e)) pure (parseIgnoreFile checkNames bytes) when (optWorktree && length optReleases /= 1) $ usageError "--worktree checks the working tree against one release, so it needs exactly one --release"- when (maybe False (\url -> not (any (`T.isInfixOf` url) ["{repo}", "{release}"])) optRepo && length optReleases /= 1) $+ when (maybe False namesOneRepo optRepo && length optReleases /= 1) $ usageError "--repo without {repo} or {release} in it names one repo, so it needs exactly one --release" token <- githubToken cores <- getNumProcessors start <- getCurrentTime- let mono = cfgMonorepo (optConfig token cores start Nothing terra (Release "rawhide" 0))- distro <- pickDistro mono optDistro+ let configFor specRef distro rel = optConfig (ConfigInputs token cores start specRef distro rel)+ mono = cfgMonorepo (configFor Nothing terra anyRelease)+ distro <- pickDistro mono optRemote optDistro+ let baseCfg = configFor Nothing distro anyRelease let specRef rel = if optWorktree then Nothing else Just (optRemote <> "/" <> releaseBranch distro rel) mkCfg rel =- let c = optConfig token cores start (specRef rel) distro rel+ let c = configFor (specRef rel) distro rel in maybe c (\url -> c {cfgRepoUrl = repoUrlFor url rel}) optRepo- releases <- resolveReleases mkCfg optReleases+ releases <- resolveReleases mono optRemote distro mkCfg optReleases - let ran = if null optOnly then checkNames else optOnly- perRelease <- forM releases $ \rel -> do+ when (cfgUpstream baseCfg && isNothing token) $+ hPutStrLn stderr "oglyad: no GITHUB_TOKEN (or `gh auth token`), GitHub lookups will hit the 60/hour limit fast"++ let ran = filter (`notElem` distroSkippedChecks distro) (if null optOnly then checkNames else optOnly)+ logPrefix rel = if length releases > 1 then T.unpack (releaseLabel rel) <> ": " else ""+ perRelease <- runApp baseCfg . forConcurrentlyN (length releases) releases $ \rel -> do let cfg = mkCfg rel- hPutStrLn stderr ("oglyad: checking " <> T.unpack (releaseLabel rel) <> " from " <> maybe "the working tree" T.unpack (cfgSpecRef cfg))- (snap, collectFindings) <- runApp cfg buildSnapshot- let findings = collectFindings <> runChecks optOnly snap+ (snap, collectFindings) <- inRelease cfg (logPrefix rel) $ do+ note ("checking " <> T.unpack (releaseLabel rel) <> " from " <> maybe "the working tree" T.unpack (cfgSpecRef cfg))+ buildSnapshot+ let findings = collectFindings <> runChecks ran snap applied | optNoIgnores = Applied findings [] [] [] | otherwise = applyIgnores checkNames ran ignores (snapIgnoreLabels snap) findings relInfo = ReleaseInfo (relName rel) (releaseLabel rel) (cfgSpecRef cfg) (snapHead snap) (repoUrl cfg Nothing) pure (rel, relInfo, applied)- -- extracted trees for commits nothing checks anymore are just disk space unless optWorktree $ pruneTrees [c | (_, ReleaseInfo {riCommit = Just c}, _) <- perRelease] now <- getCurrentTime let merged f = mergeFindings distro [(rel, f a) | (rel, _, a) <- perRelease]- -- an ignore entry is only unused if no release needed it- unused = [f | f <- merged apUnused, length (fReleases f) == length releases]+ unusedInEveryRelease = [f | f <- merged apUnused, length (fReleases f) == length releases] report = Report { rVersion = oglyadVersion,@@ -274,7 +314,7 @@ rReleases = [ri | (_, ri, _) <- perRelease], rDistro = distroName distro, rUpstream = any (\(rel, _, _) -> cfgUpstream (mkCfg rel)) perRelease,- rFindings = merged apKept <> merged apProblems <> unused,+ rFindings = merged apKept <> merged apProblems <> unusedInEveryRelease, rIgnored = merged apIgnored } LBS.putStr (render optFormat optMinSev report)
oglyad.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.0 name: oglyad-version: 0.1.4.0+version: 0.1.6.0 synopsis: repository metadata and health checker for Terra description: oglyad checks a terrapkg/packages checkout against Terra's published@@ -20,9 +20,9 @@ CHANGELOG.md README.md --- the lower bounds are what Fedora ships, which CI builds against. the upper+-- the lower bounds are what EPEL 10 ships, which CI builds against. the upper -- bounds follow the PVP from the newest versions that have been built-tested-with: GHC ==9.10.3+tested-with: GHC ==9.6.6 || ==9.10.3 source-repository head type: git@@ -43,10 +43,10 @@ -Wincomplete-record-updates -Wredundant-constraints build-depends:- , base >=4.20 && <4.21- , containers >=0.7 && <0.8- , deepseq >=1.5 && <1.6- , text >=2.1.3 && <2.2+ , base >=4.18 && <4.21+ , containers >=0.6.7 && <0.8+ , deepseq >=1.4.8 && <1.6+ , text >=2.0.2 && <2.2 , time >=1.12.2 && <1.13 -- the modules shared by the binary and the test suite. there's deliberately@@ -63,6 +63,7 @@ Oglyad.Check.FedoraOverlap Oglyad.Check.NoAutoupdate Oglyad.Check.Stale+ Oglyad.Check.Submodules Oglyad.Check.UnparsedRhai Oglyad.Check.UpstreamError Oglyad.Collect@@ -84,22 +85,22 @@ autogen-modules: Paths_oglyad build-depends:- , aeson >=2.2.3 && <2.4- , async >=2.2.6 && <2.3+ , aeson >=2.1.2 && <2.4+ , async >=2.2.5 && <2.3 , base16-bytestring >=1.0.2 && <1.1- , bytestring >=0.12.2 && <0.13- , conduit >=1.3.6 && <1.4+ , bytestring >=0.11.5 && <0.13+ , conduit >=1.3.5 && <1.4 , cryptohash-sha256 >=0.11.102 && <0.12 , directory >=1.3.8 && <1.4- , filepath >=1.5.4 && <1.6- , http-client >=0.7.19 && <0.8+ , filepath >=1.4.300 && <1.6+ , http-client >=0.7.17 && <0.8 , http-client-tls >=0.3.6 && <0.5 , http-types >=0.12.4 && <0.13- , megaparsec >=9.7 && <9.9+ , megaparsec >=9.5 && <9.9 , mtl >=2.3.1 && <2.4- , typed-process >=0.2.13 && <0.3- , vector >=0.13.2 && <0.14- , xml-conduit >=1.10.1 && <1.11+ , typed-process >=0.2.11 && <0.3+ , vector >=0.13.1 && <0.14+ , xml-conduit >=1.9.1 && <1.11 , xml-types >=0.3.8 && <0.4 executable oglyad@@ -116,6 +117,6 @@ main-is: Spec.hs ghc-options: -threaded build-depends:- , hspec >=2.11.16 && <2.12- , QuickCheck >=2.15 && <2.20+ , hspec >=2.11.9 && <2.12+ , QuickCheck >=2.14.3 && <2.20 , temporary >=1.3 && <1.4
src/Oglyad/App.hs view
@@ -1,27 +1,33 @@--- | the app monad. plain ReaderT over IO is the boring, correct starting point.--- reach for mtl classes or an effect library only once this actually hurts module Oglyad.App ( Env (..), App, HttpError (..), runApp,+ inRelease, httpGetE, forConcurrentlyN,+ processSlot,+ lookupOnce, note, oglyadVersion, ) where +import Control.Applicative ((<|>)) import Control.Concurrent (threadDelay)-import Control.Concurrent.Async (forConcurrently)-import Control.Concurrent.MVar (MVar, newMVar, withMVar)+import Control.Concurrent.Async (Async, async, forConcurrently)+import Control.Concurrent.Async qualified as Async+import Control.Concurrent.MVar (MVar, modifyMVar, newMVar, withMVar) import Control.Concurrent.QSem (QSem, newQSem, signalQSem, waitQSem) import Control.Exception (bracket_, throwIO, try)-import Control.Monad.Reader (ReaderT, ask, asks, liftIO, runReaderT)+import Control.Monad.Reader (ReaderT, ask, asks, liftIO, local, runReaderT) import Data.ByteString (ByteString) import Data.ByteString.Char8 qualified as BS8 import Data.ByteString.Lazy qualified as LBS import Data.IORef (IORef, newIORef, readIORef, writeIORef)+import Data.Map.Strict (Map)+import Data.Map.Strict qualified as Map+import Data.Maybe (fromMaybe) import Data.Text (Text) import Data.Text qualified as T import Data.Text.Encoding qualified as TE@@ -54,17 +60,19 @@ -- | GitHub's secondary rate limits punish lots of concurrent requests envGitHubSem :: QSem, -- | set once GitHub rate-limits us, so the remaining lookups are skipped- -- instead of each failing (and being reported) separately+ -- instead of each failing and being reported separately envGitHubLimited :: IORef (Maybe Text),- -- | serializes progress messages so concurrent jobs can't interleave lines- envLogLock :: MVar ()+ envLogLock :: MVar (),+ envLogPrefix :: String,+ envProcessSem :: QSem,+ envLookups :: MVar (Map Text (Async (Either HttpError Text))),+ envCacheLock :: MVar () } type App = ReaderT Env IO data HttpError- = -- | the server said slow down. not the upstream's fault, so not cached- -- and not reported per package+ = -- | not the upstream's fault, so not cached and not reported per package RateLimited Text | HttpFailed Text deriving stock (Eq, Show)@@ -76,10 +84,12 @@ userAgent :: ByteString userAgent = TE.encodeUtf8 ("oglyad/" <> oglyadVersion <> " (+https://github.com/terrapkg/oglyad)") --- | no single response may be larger than this. Fedora's compressed--- primary.xml is the biggest thing we fetch, at around 30MB+-- | Fedora's compressed primary.xml is the biggest thing we fetch, at around 30MB+maxBodyMiB :: Int+maxBodyMiB = 256+ maxBody :: Int-maxBody = 256 * 1024 * 1024+maxBody = maxBodyMiB * 1024 * 1024 runApp :: Config -> App a -> IO a runApp cfg act = do@@ -88,18 +98,23 @@ ghSem <- newQSem 6 limited <- newIORef Nothing logLock <- newMVar ()- runReaderT act (Env cfg mgr lock ghSem limited logLock)+ procSem <- newQSem (max 1 (cfgConcurrency cfg))+ lookups <- newMVar Map.empty+ cacheLock <- newMVar ()+ runReaderT act (Env cfg mgr lock ghSem limited logLock "" procSem lookups cacheLock) +inRelease :: Config -> String -> App a -> App a+inRelease cfg prefix = local (\env -> env {envConfig = cfg, envLogPrefix = prefix})+ -- | progress messages go to stderr so stdout stays clean for --json note :: String -> App () note msg = do- lock <- asks envLogLock- liftIO (withMVar lock (\_ -> hPutStrLn stderr ("oglyad: " <> msg)))+ Env {envLogLock, envLogPrefix} <- ask+ liftIO (withMVar envLogLock (\_ -> hPutStrLn stderr ("oglyad: " <> envLogPrefix <> msg))) isGitHubApi :: Text -> Bool isGitHubApi = T.isPrefixOf "https://api.github.com/" --- | GET a url, throwing on non-2xx httpGet :: Text -> App LBS.ByteString httpGet url = do Env {..} <- ask@@ -110,13 +125,12 @@ req' = req { requestHeaders = ("User-Agent", userAgent) : auth <> requestHeaders req,- -- never forward the token if a redirect goes somewhere else shouldStripHeaderOnRedirect = (== hAuthorization) } fetch = withResponse req' envHttp $ \resp -> do body <- brReadSome (responseBody resp) (maxBody + 1) if LBS.length body > fromIntegral maxBody- then throwIO (userError ("response from " <> T.unpack url <> " is over 256MiB"))+ then throwIO (userError ("response from " <> T.unpack url <> " is over " <> show maxBodyMiB <> "MiB")) else pure body liftIO $ if@@ -124,9 +138,8 @@ | isGitHubApi url -> bracket_ (waitQSem envGitHubSem) (signalQSem envGitHubSem) fetch | otherwise -> fetch --- | GET a url, with failures as a short readable Left instead of an--- exception. a rate limit with a short Retry-After gets one retry. a GitHub--- rate limit beyond that stops all further GitHub requests for this run+-- | a short rate limit gets one retry. a longer GitHub one stops all further+-- GitHub requests for this run httpGetE :: Text -> App (Either HttpError LBS.ByteString) httpGetE url = do env@Env {..} <- ask@@ -142,7 +155,7 @@ Left (HttpExceptionRequest _ (StatusCodeException resp _)) -> do wait <- rateLimitWait resp case wait of- Just secs | n == 1 && secs <= 60 -> do+ Just secs | n == 1 && secs <= longestRetryWait -> do threadDelay ((secs + 1) * 1_000_000) attempt env (n + 1) Just _ -> do@@ -152,11 +165,11 @@ Nothing -> pure (Left (HttpFailed ("HTTP " <> T.pack (show (statusCode (responseStatus resp))) <> " from " <> url))) Left (HttpExceptionRequest _ content) -> pure (Left (HttpFailed (T.pack (show content) <> " from " <> url))) Left (InvalidUrlException u why) -> pure (Left (HttpFailed ("invalid url " <> T.pack u <> ": " <> T.pack why)))- host = T.takeWhile (/= '/') (T.drop 8 url)+ host = T.takeWhile (/= '/') (T.drop (T.length "https://") url)+ longestRetryWait = 60 --- | if this response is a rate limit, how many seconds until it lifts.--- GitHub signals its primary limit with a 403 and x-ratelimit-remaining: 0,--- and its secondary limit (too many concurrent requests) with a 403 or 429+-- | seconds until a rate limit lifts. GitHub signals its primary limit with a+-- 403 and x-ratelimit-remaining: 0, and its secondary limit with a 403 or 429 -- plus retry-after rateLimitWait :: Response () -> IO (Maybe Int) rateLimitWait resp = do@@ -170,12 +183,9 @@ pure $ if | status == 429 || (status == 403 && (exhausted || retryAfter /= Nothing)) ->- Just (maybe (maybe 3600 id untilReset) id retryAfter)+ Just (fromMaybe 3600 (retryAfter <|> untilReset)) | otherwise -> Nothing --- | run an action over a list with at most n in flight at once.--- 'forConcurrently' lives in IO, so we grab the env and re-run App inside each--- thread. that dance is exactly what MonadUnliftIO (from unliftio) automates forConcurrentlyN :: Int -> [a] -> (a -> App b) -> App [b] forConcurrentlyN n xs f = do env <- ask@@ -183,3 +193,19 @@ sem <- newQSem (max 1 n) forConcurrently xs $ \x -> bracket_ (waitQSem sem) (signalQSem sem) (runReaderT (f x) env)++processSlot :: App (IO a -> IO a)+processSlot = do+ sem <- asks envProcessSem+ pure (bracket_ (waitQSem sem) (signalQSem sem))++lookupOnce :: Text -> App (Either HttpError Text) -> App (Either HttpError Text)+lookupOnce key act = do+ env <- ask+ liftIO $ do+ running <- modifyMVar (envLookups env) $ \m -> case Map.lookup key m of+ Just inFlight -> pure (m, inFlight)+ Nothing -> do+ started <- async (runReaderT act env)+ pure (Map.insert key started m, started)+ Async.wait running
src/Oglyad/Check.hs view
@@ -1,4 +1,3 @@--- | the check registry. adding a check = write a module, add a line here module Oglyad.Check ( allChecks, checkNames,@@ -16,6 +15,7 @@ import Oglyad.Check.FedoraOverlap qualified as FedoraOverlap import Oglyad.Check.NoAutoupdate qualified as NoAutoupdate import Oglyad.Check.Stale qualified as Stale+import Oglyad.Check.Submodules qualified as Submodules import Oglyad.Check.UnparsedRhai qualified as UnparsedRhai import Oglyad.Check.UpstreamError qualified as UpstreamError import Oglyad.Types@@ -28,6 +28,7 @@ ("fedora-overlap", FedoraOverlap.check), ("no-autoupdate", NoAutoupdate.check), ("stale", Stale.check),+ ("submodules", Submodules.check), ("unparsed-rhai", UnparsedRhai.check), ("upstream-error", UpstreamError.check) ]@@ -35,8 +36,8 @@ checkNames :: [Text] checkNames = map fst allChecks --- | validate a check name from the command line. a typo has to be an error,--- otherwise `--only buildlag` quietly runs nothing and reports all clear+-- | a typo has to be an error, otherwise `--only buildlag` quietly runs+-- nothing and reports all clear parseCheckName :: String -> Either String Text parseCheckName raw | name `elem` checkNames = Right name@@ -45,12 +46,11 @@ where name = T.pack raw --- | run the named checks (all of them when the list is empty) over every package runChecks :: [Text] -> Snapshot -> [Finding]-runChecks only snap =+runChecks names snap = [ f | (name, chk) <- allChecks,- null only || name `elem` only,+ name `elem` names, pkg <- Map.elems (snapPackages snap), f <- chk snap pkg ]
src/Oglyad/Check/BuildLag.hs view
@@ -1,4 +1,3 @@--- | what's merged in git doesn't match what's published module Oglyad.Check.BuildLag (check) where import Oglyad.Types@@ -6,19 +5,17 @@ check :: Check check _ p@Package {..} = case (pkgSpecEVR, pkgRepoBuild) of- -- the repo fetch failed, which is already a collect error. claiming every- -- package in it was never published would bury that under noise+ -- the failed repo fetch is already a collect error, so don't bury it under+ -- "never published" noise _ | not pkgRepoChecked -> [] (Just spec, Nothing) -> [finding' Error ("never published (spec is at " <> renderEVR spec <> ")")] (Just spec, Just RepoBuild {rbEVR = repo}) | spec == repo -> [] | otherwise -> case compareEVR spec repo of- -- merged but not built: the build failed or is still queued GT -> [finding' Warn ("spec is at " <> renderEVR spec <> " but repo has " <> renderEVR repo)]- -- the repo is newer than git, so the monorepo checkout is behind LT -> [finding' Info ("repo has " <> renderEVR repo <> " but spec is at " <> renderEVR spec <> ", is the checkout up to date?")]- -- same version spelled differently, e.g. 1.05 vs 1.5+ -- spelled differently, e.g. 1.05 vs 1.5 EQ -> [] _ -> [] where
src/Oglyad/Check/Drift.hs view
@@ -1,4 +1,3 @@--- | upstream has a newer version than the spec module Oglyad.Check.Drift (check, drift) where import Data.Char (isAlphaNum)@@ -7,12 +6,9 @@ import Oglyad.Types import Oglyad.Version (compareUpstream) --- | (upstream version, spec version, how upstream compares), for packages--- whose version is meant to follow upstream releases. nightlies (a `nightly`--- or `quaterdaily` label in anda.hcl) build from a branch head and only--- borrow the last release as a base, so the newest release isn't something--- they lag behind. `weekly` isn't one of them: those are mostly releases--- (fonts, nvidia) checked less often+-- | (upstream, spec, ordering) for packages meant to follow upstream releases.+-- nightlies only borrow the last release as a base, so they can't lag behind+-- it. `weekly` packages still count, those are mostly releases checked less often drift :: Package -> Maybe (Text, Text, Ordering) drift p | pkgNightly p = Nothing@@ -24,26 +20,25 @@ spec <- evrVersion <$> pkgSpecEVR pure (up, spec, compareUpstream up spec) --- pkgUpstreamVer is only populated with --upstream, so this check is silent--- by default check :: Check check _ p = case compared p of- -- usually a crop(1) meant for a "v" prefix on a tag that doesn't have one,- -- which turns 6.0.1 into .0.1. a bug in the script, nightly or not+ -- usually a crop(1) meant for a "v" prefix the tag doesn't have, turning+ -- 6.0.1 into .0.1 Just (up, _, _)- | maybe True (not . isAlphaNum . fst) (T.uncons up) ->+ | not (startsWithAlphaNum up) -> [finding' Warn ("update.rhai would set the version to \"" <> up <> "\", which doesn't start with a letter or digit")] _ -> case drift p of Just r -> compareFindings r Nothing -> [] where finding' = finding "drift" p+ startsWithAlphaNum = maybe False (isAlphaNum . fst) . T.uncons+ isSnapshot = T.any (== '^') compareFindings = \case (up, spec, GT) -> [finding' Warn ("upstream has " <> up <> ", spec is at " <> spec)] (up, spec, LT)- -- spec ahead of upstream. snapshots (Version: 1.2^20251101git...)- -- built after the 1.2 release always look like this and are fine.- -- anything else usually means a tag that normalizes badly- | T.any (== '^') spec -> []+ -- a snapshot like 1.2^20251101git built after the 1.2 release always+ -- sorts ahead of it+ | isSnapshot spec -> [] | otherwise -> [finding' Info ("spec " <> spec <> " is ahead of upstream " <> up <> ", check tag normalization")] _ -> []
src/Oglyad/Check/DuplicateName.hs view
@@ -1,8 +1,6 @@--- | two monorepo directories building the same source package into the same--- repo. only one build can be live in a repo, so they overwrite each other,--- and nothing (including build-lag) can tell which one a published rpm came--- from. sometimes deliberate (a bootstrap build, a 32-bit variant), sometimes--- a copy-pasted Name: that was never changed+-- | only one build can be live in a repo, so two directories with the same+-- Name: overwrite each other and nothing can tell which one a published rpm+-- came from module Oglyad.Check.DuplicateName (check) where import Data.Map.Strict qualified as Map
src/Oglyad/Check/FedoraOverlap.hs view
@@ -1,14 +1,13 @@--- | a package should live in Terra or Fedora, not both module Oglyad.Check.FedoraOverlap (check) where import Data.Set qualified as Set import Oglyad.Types --- | subrepos (extras, mesa, multimedia, nvidia) exist to ship replacements--- for Fedora packages, so overlap there is the whole point+-- | subrepos (extras, mesa, multimedia, nvidia) exist to replace base distro+-- packages check :: Check check Snapshot {..} p@Package {..} | Just _ <- pkgSubrepo = []- | pkgName `Set.member` snapFedora =- [finding "fedora-overlap" p Warn "also shipped by Fedora"]+ | pkgName `Set.member` snapBase =+ [finding "fedora-overlap" p Warn ("also shipped by " <> cfgBaseName snapConfig)] | otherwise = []
src/Oglyad/Check/NoAutoupdate.hs view
@@ -1,4 +1,3 @@--- | packages with no update.rhai have to be bumped by hand module Oglyad.Check.NoAutoupdate (check) where import Oglyad.Types
src/Oglyad/Check/Stale.hs view
@@ -1,4 +1,3 @@--- | packages that haven't been built in a long time module Oglyad.Check.Stale (check) where import Data.Text (Text)@@ -7,9 +6,8 @@ import Oglyad.Check.Drift (drift) import Oglyad.Types --- | a stable package nobody touched for a year might be totally fine, so an--- old build alone is Info. it only becomes a Warn when upstream has moved on--- (needs --upstream) and the package is still sitting on the old version+-- | an old build alone might be totally fine, so it only warns once upstream+-- has moved on check :: Check check Snapshot {..} p@Package {..} = case pkgRepoBuild of Nothing -> [] -- never published is build-lag's job
+ src/Oglyad/Check/Submodules.hs view
@@ -0,0 +1,25 @@+module Oglyad.Check.Submodules (check) where++import Data.Text qualified as T+import Oglyad.Types++check :: Check+check _ p = map describe (pkgSubmoduleEvents p)+ where+ describe = \case+ SubmoduleAdded {..} ->+ finding "submodules" p Warn $+ "upstream added the submodule "+ <> sePath+ <> " ("+ <> seUrl+ <> " at "+ <> T.take 12 seCommit+ <> "). add %global "+ <> seGlobal+ <> " with a Source and %submodule_setup line, or pass it to rpm.submodules() as ignore"+ SubmoduleDropped {..} ->+ finding "submodules" p Warn $+ "upstream dropped the submodule for %global "+ <> seGlobal+ <> ". remove the global, its Source and its %submodule_setup line"
src/Oglyad/Check/UnparsedRhai.hs view
@@ -1,5 +1,3 @@--- | reports update.rhai shapes our parser doesn't understand yet. this is your--- to-do list for Oglyad.Source.Rhai module Oglyad.Check.UnparsedRhai (check) where import Data.Maybe (isJust)@@ -8,19 +6,17 @@ import Oglyad.Types check :: Check-check _ p@Package {..} =+check Snapshot {snapConfig} p@Package {..} = broken <> case pkgUpstream of- -- anda ran it for us, so there's an answer (or an upstream-error)+ -- anda ran it, so there's already an answer or an upstream-error Just (Unknown _) | isJust pkgUpstreamVer || isJust pkgUpstreamErr -> []+ Just (Unknown _) | not (cfgRolling snapConfig || pkgUpdBranch) -> [] Just (Unknown raw) ->- [finding "unparsed-rhai" p Info ("can't parse update.rhai: " <> T.take 80 (head' (T.lines raw)))]+ [finding "unparsed-rhai" p Info ("can't parse update.rhai: " <> T.take 80 (T.takeWhile (/= '\n') raw))] _ -> [] where- -- not our parser's limits but a script Rhai itself rejects, so- -- `anda update` fails on it every run+ -- Rhai itself rejects these, so `anda update` fails on them every run broken = case tokenize <$> pkgRhai of Just (Just toks) | not (balanced toks) -> [finding "unparsed-rhai" p Warn "update.rhai has unbalanced brackets, so anda can't run it"] _ -> []- head' (l : _) = l- head' [] = ""
src/Oglyad/Check/UpstreamError.hs view
@@ -1,5 +1,3 @@--- | the update.rhai source parsed fine but asking upstream failed. usually a--- renamed or deleted repo, which means `anda update` is failing on it too module Oglyad.Check.UpstreamError (check) where import Oglyad.Types
src/Oglyad/Collect.hs view
@@ -1,11 +1,12 @@--- | the imperative shell: gather every data source into one 'Snapshot' module Oglyad.Collect ( buildSnapshot, ) where +import Control.Applicative ((<|>)) import Control.Concurrent.Async (concurrently)-import Control.Monad (forM, when)+import Control.Concurrent.MVar (withMVar)+import Control.Monad (forM, join) import Control.Monad.Reader (ask, asks, liftIO, runReaderT) import Data.Either (partitionEithers) import Data.List (nub, partition)@@ -14,21 +15,22 @@ import Data.Maybe (fromMaybe, isJust, isNothing, mapMaybe) import Data.Text (Text) import Data.Text qualified as T-import Data.Time (UTCTime, diffUTCTime, getCurrentTime, nominalDay)-import Oglyad.App (App, Env (..), HttpError (..), forConcurrentlyN, note)-import Oglyad.Source.Anda (projectNames, updateProject, withScratchCopy)-import Oglyad.Source.Cache (CacheEntry (..), UpstreamCache, loadJson, lookupFresh, saveJson)+import Data.Time (NominalDiffTime, UTCTime, diffUTCTime, getCurrentTime, nominalDay)+import Oglyad.App (App, Env (..), HttpError (..), forConcurrentlyN, lookupOnce, note, processSlot)+import Oglyad.Source.Anda (projectNames, reportSupported, updateProject, withScratchCopy)+import Oglyad.Source.Cache (AndaEntry (..), CacheEntry (..), UpstreamCache, loadJson, lookupFresh, saveJson) import Oglyad.Source.Monorepo import Oglyad.Source.Repodata (fetchRepo)-import Oglyad.Source.Rhai (parseUpdateRhaiWith, tokenize)+import Oglyad.Source.Rhai (parseUpdateRhaiWith, tokenize, tracksSubmodules) import Oglyad.Source.RhaiEval (Simulated (..), simulateVersion) import Oglyad.Source.Upstream (latestVersion) import Oglyad.Types import System.Directory (findExecutable, makeAbsolute) import System.FilePath ((</>))+import System.Info qualified --- | collect everything. problems along the way become findings instead of--- crashes, so one broken spec doesn't take down the whole run.+-- | problems along the way become findings instead of crashes, so one broken+-- spec doesn't take down the whole run. -- -- the monorepo work (rpmspec, git) is all local cpu and the repodata work is -- all network, so the two halves run at the same time@@ -36,43 +38,40 @@ buildSnapshot = do cfg@Config {..} <- asks envConfig now <- liftIO getCurrentTime- -- specs come from a release branch extracted into the cache, or straight- -- from the working tree with --worktree tree <- liftIO $ case cfgSpecRef of Nothing -> Right . (cfgMonorepo,"HEAD",) <$> headCommit cfgMonorepo Just ref -> fmap (\(dir, sha) -> (dir, sha, Just sha)) <$> checkoutRef cfgMonorepo ref let (root, logRef, hd, treeErrs) = case tree of Right (r, l, h) -> (Just r, l, h, [])- Left err -> (Nothing, "HEAD", Nothing, [collectErr (maybe "monorepo" id cfgSpecRef) err])+ Left err -> (Nothing, "HEAD", Nothing, [collectErr (fromMaybe "monorepo" cfgSpecRef) err]) (dirErrs, dirs) <- partitionEithers <$> liftIO (maybe (pure []) (`findPackageDirs` cfgRoot) root) - -- the main repo, every subrepo some package asked for, and Fedora- let terraRepos = [(s, repoUrl cfg s) | s <- Nothing : nub (map Just (mapMaybe pdSubrepo dirs))]+ let subrepos = if cfgSubrepos then nub (map Just (mapMaybe pdSubrepo dirs)) else []+ terraRepos = [(s, repoUrl cfg s) | s <- Nothing : subrepos]+ publishedTo p = if cfgSubrepos then pkgSubrepo p else Nothing env <- ask- ((specErrs, pkgs0), (terra, fedora)) <-+ ((specErrs, pkgs0), (terra, base)) <- liftIO $ concurrently- (runReaderT (readPackages (maybe cfgMonorepo id root) logRef hd dirs) env)- (runReaderT (fetchRepos terraRepos cfgFedoraRepos) env)+ (runReaderT (readPackages (fromMaybe cfgMonorepo root) logRef hd dirs) env)+ (runReaderT (fetchRepos terraRepos cfgBaseRepos) env) let (repoErrs, repos) = partitionEithers terra fetched = Map.fromList repos pkgs1 = [ p- { pkgRepoBuild = Map.lookup (pkgSubrepo p) fetched >>= Map.lookup (pkgName p),- pkgRepoChecked = Map.member (pkgSubrepo p) fetched+ { pkgRepoBuild = Map.lookup (publishedTo p) fetched >>= Map.lookup (pkgName p),+ pkgRepoChecked = Map.member (publishedTo p) fetched } | p <- pkgs0 ]- (upErrs, pkgs2) <- if cfgUpstream then withUpstreams now pkgs1 else pure ([], pkgs1)- (andaErrs, pkgs) <- case root of- Just r | cfgUpstream -> withAnda r now pkgs2- _ -> pure ([], pkgs2)+ (upErrs, withUpstream) <- if cfgUpstream then upstreamVersions now root pkgs1 else pure ([], id)+ let pkgs = map withUpstream pkgs1 let snap = Snapshot { snapPackages = Map.fromList [(pkgPath p, p) | p <- pkgs],- snapFedora = foldMap Map.keysSet fedora,+ snapBase = foldMap Map.keysSet base, snapNow = now, snapConfig = cfg, snapHead = hd,@@ -82,48 +81,57 @@ [ (collectErr (T.pack path) ("can't read package dir: " <> err)) {fPath = Just (T.pack path)} | (path, err) <- dirErrs ]- pure (snap, treeErrs <> unreadable <> specErrs <> repoErrs <> upErrs <> andaErrs)+ pure (snap, treeErrs <> unreadable <> specErrs <> repoErrs <> upErrs) --- | every repo at once. Terra failures become findings, Fedora ones just mean--- no overlap data+upstreamVersions :: UTCTime -> Maybe FilePath -> [Package] -> App ([Finding], Package -> Package)+upstreamVersions now root pkgs = do+ env <- ask+ ((upErrs, fromApis), (andaErrs, fromAnda)) <-+ liftIO $+ concurrently+ (runReaderT (withUpstreams now pkgs) env)+ (runReaderT (maybe (pure ([], id)) (\r -> withAnda r now pkgs) root) env)+ pure (upErrs <> andaErrs, fromAnda . fromApis)++-- | Terra failures become findings, base distro ones just mean no overlap+-- data fetchRepos :: [(Maybe Text, Text)] -> [Text] -> App ([Either Finding (Maybe Text, Map Text RepoBuild)], [Map Text RepoBuild])-fetchRepos terraRepos fedoraRepos = do- let jobs = map Left terraRepos <> map Right fedoraRepos+fetchRepos terraRepos baseRepos = do+ let jobs = map Left terraRepos <> map Right baseRepos results <- forConcurrentlyN (length jobs) jobs $ \case Left (sub, url) -> Left . either (Left . collectErr url . T.pack) (\b -> Right (sub, b)) <$> fetchRepo url Right url -> Right . either (const Map.empty) id <$> fetchRepo url pure (partitionEithers results) --- | evaluate every spec under root, reusing cached evaluations for unchanged--- packages. logRef and hd say which commit's history to read commit times from readPackages :: FilePath -> Text -> Maybe Text -> [PkgDir] -> App ([Finding], [Package])-readPackages root0 logRef hd dirs = do+readPackages root0 logRef headSha dirs = do Config {..} <- asks envConfig root <- liftIO (makeAbsolute root0)- -- one cache file per dist, so cron jobs checking different releases don't- -- keep evicting each other's entries- let cacheFile = "specs" <> T.unpack cfgDist <> ".json"+ -- one cache file per release, so cron jobs checking different releases+ -- don't keep evicting each other's entries+ let cacheFile = "specs" <> T.unpack cfgCacheTag <> ".json" (commits, (macros, cache)) <- liftIO $ concurrently- (commitTimes cfgMonorepo cfgRoot logRef hd cfgDist (map pdPath dirs))+ (commitTimes cfgMonorepo cfgRoot logRef headSha cfgCacheTag (map pdPath dirs)) ((,) <$> macroFingerprint <*> (fromMaybe Map.empty <$> loadJson cacheFile)) keyed <- forConcurrentlyN cfgConcurrency dirs $ \d -> liftIO $ do- key <- specFingerprint macros cfgDist root (pdPath d)+ key <- specFingerprint macros cfgMacros root (pdPath d) pure (key, d) let misses = length [() | (k, _) <- keyed, isNothing (Map.lookup k (cache :: Map Text (Either Text SpecInfo)))] note ("evaluating " <> show (length dirs) <> " specs, " <> show (length dirs - misses) <> " unchanged since last run") -- without rpmspec, cached evaluations still work. everything else is -- skipped and reported once, rather than as one error per package hasRpmspec <- liftIO (isJust <$> findExecutable "rpmspec")+ inSlot <- processSlot results <- forConcurrentlyN cfgConcurrency keyed $ \(key, d) -> liftIO $ do case (Map.lookup key cache, pdSpec d) of (Just cached, _) -> pure (Evaluated key d cached) (Nothing, Just spec)- | hasRpmspec -> Evaluated key d <$> querySpec cfgRelease cfgDist (root </> spec)+ | hasRpmspec -> Evaluated key d <$> inSlot (querySpec cfgMacros (root </> spec)) | otherwise -> pure Skipped (Nothing, Nothing) -> pure (NoSpec d) -- keep only entries for packages that still exist, so the file can't grow forever@@ -135,22 +143,27 @@ ] (errs, pkgs) = partitionEithers $- [toPackage commits k d (Just i) | Evaluated k d i <- results]+ [toPackage commits k d (Just i) | Evaluated k d i <- results, not (failedForThisHost d i)] <> [toPackage commits "" d Nothing | NoSpec d <- results] pure (missing <> errs, pkgs) --- | what happened to one package dir's spec data SpecResult = Evaluated Text PkgDir (Either Text SpecInfo) | NoSpec PkgDir | Skipped --- | last commit per package dir, reused as long as the commit hasn't moved.--- uncommitted changes can't change any "last commit", so the commit is the--- whole key. one cache file per dist, since each release reads its own branch+failedForThisHost :: PkgDir -> Either Text SpecInfo -> Bool+failedForThisHost PkgDir {pdArches} = \case+ Left err -> "No compatible architectures found for build" `T.isInfixOf` err && not buildsHere+ Right _ -> False+ where+ buildsHere = null pdArches || T.pack System.Info.arch `elem` pdArches++-- | uncommitted changes can't change any "last commit", so the commit is the+-- whole cache key. one cache file per release, since each reads its own branch commitTimes :: FilePath -> FilePath -> Text -> Maybe Text -> Text -> [FilePath] -> IO (Map FilePath UTCTime)-commitTimes mono pkgRoot ref hd dist dirs = do- let file = "commits" <> T.unpack dist <> ".json"+commitTimes mono pkgRoot ref hd tag dirs = do+ let file = "commits" <> T.unpack tag <> ".json" cached <- loadJson file case (hd, cached) of (Just h, Just (h', m)) | h == h', all (`Map.member` m) dirs -> pure m@@ -170,45 +183,39 @@ pkgPath = pdPath, pkgSubrepo = pdSubrepo, pkgNightly = pdNightly,+ pkgUpdBranch = pdUpdBranch, pkgSpecPath = pdSpec, pkgFingerprint = key, pkgSpecEVR = Just siEVR, pkgRepoBuild = Nothing, pkgLastCommit = Map.lookup pdPath commits,- -- an update.rhai with nothing but comments in it updates nothing- pkgUpstream = parseUpdateRhaiWith pdVersionMacros <$> (pdUpdateRhai >>= nonEmpty),+ pkgUpstream = parseUpdateRhaiWith pdVersionMacros <$> (pdUpdateRhai >>= unlessOnlyComments), pkgRhai = pdUpdateRhai, pkgUpstreamVer = Nothing, pkgUpstreamErr = Nothing,+ pkgSubmoduleEvents = [], pkgRepoChecked = False } where firstLine t = let l = T.takeWhile (/= '\n') t in if T.null l then "(no output)" else l- nonEmpty script = case tokenize script of+ unlessOnlyComments script = case tokenize script of Just [] -> Nothing _ -> Just script pathErr msg = (collectErr (T.pack pdPath) msg) {fPath = Just (T.pack pdPath)} --- | fill in upstream versions, from the cache when it's fresh enough.------ crates.io allows one request a second, so its lookups run in their own+-- | crates.io allows one request a second, so its lookups run in their own -- serial queue next to everything else. inside the shared job pool, a cluster -- of crates packages would hold every slot while waiting their turn-withUpstreams :: UTCTime -> [Package] -> App ([Finding], [Package])+withUpstreams :: UTCTime -> [Package] -> App ([Finding], Package -> Package) withUpstreams now pkgs = do- Config {..} <- asks envConfig- when (isNothing cfgGitHubToken) $- note "no GITHUB_TOKEN (or `gh auth token`), GitHub lookups will hit the 60/hour limit fast"+ cfg@Config {..} <- asks envConfig cache <- liftIO (fromMaybe Map.empty <$> loadJson "upstream.json") :: App UpstreamCache let wanted = [p | p <- pkgs, Just src <- [pkgUpstream p], lookable src]- -- anything fetched since this run started is fresh, so checking several- -- releases asks each upstream once even with --cache-hours 0- maxAge = max cfgCacheFor (diffUTCTime now cfgRunStart)- fresh p = pkgUpstream p >>= \src -> lookupFresh maxAge now src cache+ fresh p = pkgUpstream p >>= \src -> lookupFresh (cacheMaxAge cfg now) now src cache (crates, others) = partition (isCrates . pkgUpstream) wanted lookup1 p = case (fresh p, pkgUpstream p) of (Just e, _) -> pure (pkgPath p, either (Left . HttpFailed) Right (ceResult e))- (Nothing, Just src) -> (pkgPath p,) <$> latestVersion src+ (Nothing, Just src) -> (pkgPath p,) <$> lookupOnce (cacheKey src) (latestVersion src) (Nothing, Nothing) -> pure (pkgPath p, Left (HttpFailed "no source")) note ("querying upstreams for " <> show (length wanted) <> " packages, " <> show (length (filter (isJust . fresh) wanted)) <> " cached") env <- ask@@ -222,14 +229,12 @@ -- only cache successes. errors are often transient (rate limits, timeouts) new = Map.fromList- [ (T.pack (show src), CacheEntry now (Right v))+ [ (cacheKey src, CacheEntry now (Right v)) | p <- wanted, isNothing (fresh p), Just src <- [pkgUpstream p], Just (Right v) <- [Map.lookup (pkgPath p) byPath] ]- -- forget answers nobody has asked for in a month- recent = Map.filter (\e -> diffUTCTime now (ceFetched e) < 30 * nominalDay) cache limited = [why | Left (RateLimited why) <- Map.elems byPath] rateErr = case limited of [] -> []@@ -238,19 +243,18 @@ { fSeverity = Warn } ]- liftIO (saveJson "upstream.json" (new <> recent))+ saveSharedCache "upstream.json" now new pure ( rateErr,- [ case Map.lookup (pkgPath p) byPath of- Just (Right v) -> p {pkgUpstreamVer = replay p v}- Just (Left (HttpFailed e)) -> p {pkgUpstreamErr = Just e}- -- rate limits are reported once above, not as a broken upstream- Just (Left (RateLimited _)) -> p- Nothing -> p- | p <- pkgs- ]+ \p -> case Map.lookup (pkgPath p) byPath of+ Just (Right v) -> p {pkgUpstreamVer = replay p v}+ Just (Left (HttpFailed e)) -> p {pkgUpstreamErr = Just e}+ -- rate limits are reported once above, not as a broken upstream+ Just (Left (RateLimited _)) -> p+ Nothing -> p ) where+ cacheKey src = T.pack (show src) -- run the script's guards and edits over the raw value, so a tag the -- script would skip (or crop) doesn't show up as drift replay p v = case (pkgUpstream p, pkgRhai p) of@@ -269,66 +273,87 @@ Just (Crates _ _) -> True _ -> False --- | run the real anda for scripts oglyad couldn't follow, on a throwaway--- copy of the tree, and read the Version it leaves in each spec. results are--- cached like upstream answers, keyed by the package's fingerprint-withAnda :: FilePath -> UTCTime -> [Package] -> App ([Finding], [Package])+withAnda :: FilePath -> UTCTime -> [Package] -> App ([Finding], Package -> Package) withAnda root now pkgs = do- Config {..} <- asks envConfig- let wanted = [p | p@Package {pkgUpstream = Just (Unknown _), pkgSpecPath = Just _} <- pkgs]- cacheFile = "anda" <> T.unpack cfgDist <> ".json"- maxAge = max cfgCacheFor (diffUTCTime now cfgRunStart)- cache <- liftIO (fromMaybe Map.empty <$> loadJson cacheFile) :: App (Map Text CacheEntry)+ cfg@Config {..} <- asks envConfig+ let wanted =+ [ p+ | p@Package {pkgSpecPath = Just _} <- pkgs,+ unfollowable p || maybe False tracksSubmodules (pkgRhai p),+ cfgRolling || pkgUpdBranch p+ ]+ cacheFile = "anda" <> T.unpack cfgCacheTag <> ".json"+ cache <- liftIO (fromMaybe Map.empty <$> loadJson cacheFile) :: App (Map Text AndaEntry) let fresh p = case Map.lookup (pkgFingerprint p) cache of- Just e | diffUTCTime now (ceFetched e) <= maxAge -> Just e+ Just e | diffUTCTime now (aeFetched e) <= cacheMaxAge cfg now -> Just e _ -> Nothing misses = [p | p <- wanted, isNothing (fresh p)] hasAnda <- liftIO (isJust <$> findExecutable "anda")+ andaReports <- if hasAnda && not (null misses) then liftIO reportSupported else pure False (problems, ran) <- if | null misses -> pure ([], Map.empty) | not hasAnda -> do- note ("anda not found, so " <> show (length misses) <> " update.rhai scripts oglyad can't follow are skipped")+ note ("anda not found, so " <> show (length misses) <> " update.rhai scripts that need anda are skipped") pure ([], Map.empty)+ | not andaReports ->+ pure ([(collectErr "anda" ("anda is too old for `anda update --report`, so " <> T.pack (show (length misses)) <> " update.rhai scripts that need anda are skipped. update anda")) {fSeverity = Warn}], Map.empty) | otherwise -> do- note ("running anda update for " <> show (length wanted) <> " scripts oglyad can't follow, " <> show (length wanted - length misses) <> " cached")+ note ("running anda update for " <> show (length wanted) <> " scripts that need anda, " <> show (length wanted - length misses) <> " cached") env <- ask+ inSlot <- processSlot r <- liftIO $ withScratchCopy root $ \copy -> projectNames copy >>= \case Left e -> pure (Left e) Right names -> fmap Right . flip runReaderT env . forConcurrentlyN cfgConcurrency misses $ \p ->- liftIO $+ liftIO . inSlot $ (pkgPath p,) <$> case (Map.lookup (pkgPath p) names, pkgSpecPath p) of (Just name, Just spec) -> updateProject copy cfgGitHubToken cfgBranch name >>= \case Left e -> pure (Left e)- Right () ->- either (Left . ("rpmspec failed on the updated spec: " <>)) (Right . evrVersion . siEVR)- <$> querySpec cfgRelease cfgDist (copy </> spec)+ Right events ->+ either (Left . ("rpmspec failed on the updated spec: " <>)) (\i -> Right (evrVersion (siEVR i), events))+ <$> querySpec cfgMacros (copy </> spec) _ -> pure (Left "anda doesn't list this project")- pure $ case join' r of+ pure $ case join r of Left e -> ([(collectErr "anda" e) {fSeverity = Warn}], Map.empty) Right xs -> ([], Map.fromList xs)- let answer p = maybe (Map.lookup (pkgPath p) ran) (Just . ceResult) (fresh p)+ let answer p = maybe (Map.lookup (pkgPath p) ran) (\e -> Just ((,aeEvents e) <$> aeResult e)) (fresh p) new = Map.fromList- [ (pkgFingerprint p, CacheEntry now (Right v))+ [ (pkgFingerprint p, AndaEntry now (Right v) events) | p <- misses,- Just (Right v) <- [Map.lookup (pkgPath p) ran]+ Just (Right (v, events)) <- [Map.lookup (pkgPath p) ran] ]- recent = Map.filter (\e -> diffUTCTime now (ceFetched e) < 30 * nominalDay) cache- liftIO (saveJson cacheFile (new <> recent))+ liftIO (saveJson cacheFile (new <> Map.filter ((< 30 * nominalDay) . diffUTCTime now . aeFetched) cache)) pure ( problems,- [ case answer p of- Just (Right v) -> p {pkgUpstreamVer = Just v}- Just (Left e) -> p {pkgUpstreamErr = Just ("update.rhai failed under anda: " <> e)}- Nothing -> p- | p <- pkgs- ]+ \p -> case answer p of+ Just (Right (v, events))+ | unfollowable p -> p {pkgUpstreamVer = Just v, pkgSubmoduleEvents = events}+ | otherwise -> p {pkgSubmoduleEvents = events}+ Just (Left e) -> p {pkgUpstreamErr = pkgUpstreamErr p <|> Just ("update.rhai failed under anda: " <> e)}+ Nothing -> p ) where- join' = either Left id+ unfollowable p = case pkgUpstream p of+ Just (Unknown _) -> True+ _ -> False++-- | anything fetched since this run started is fresh, so checking several+-- releases asks each upstream once even with --cache-hours 0+cacheMaxAge :: Config -> UTCTime -> NominalDiffTime+cacheMaxAge Config {..} now = max cfgCacheFor (diffUTCTime now cfgRunStart)++fetchedThisMonth :: UTCTime -> CacheEntry -> Bool+fetchedThisMonth now e = diffUTCTime now (ceFetched e) < 30 * nominalDay++saveSharedCache :: FilePath -> UTCTime -> Map Text CacheEntry -> App ()+saveSharedCache file now new = do+ lock <- asks envCacheLock+ liftIO . withMVar lock $ \_ -> do+ onDisk <- fromMaybe Map.empty <$> loadJson file+ saveJson file (new <> Map.filter (fetchedThisMonth now) onDisk) collectErr :: Text -> Text -> Finding collectErr what msg = Finding {fCheck = "collect", fPackage = what, fPath = Nothing, fSeverity = Error, fMessage = msg, fReleases = []}
src/Oglyad/Ignore.hs view
@@ -1,14 +1,3 @@--- | silencing findings that are known and intentional.------ two sources, both matched by check name:------ * an ignore file (--ignore-file), where every entry names a package by--- monorepo path or package name and must give a reason--- * an `oglyad_ignore = "check, other-check"` label in a package's--- anda.hcl, so the exception lives next to the package------ ignored findings don't count toward the exit code. ignore-file entries that--- no longer match anything are reported, so stale exceptions get cleaned up module Oglyad.Ignore ( IgnoreRule (..), Target (..),@@ -29,12 +18,9 @@ import Data.Text qualified as T import Oglyad.Types --- | what an ignore rule points at data Target- = -- | a monorepo directory, e.g. anda/langs/zig/master-bootstrap- ByPath Text- | -- | every directory building this package name- ByName Text+ = ByPath Text+ | ByName Text deriving stock (Eq, Show) data IgnoreRule = IgnoreRule@@ -64,7 +50,6 @@ then fail ("entry for " <> T.unpack check <> " has an empty reason. say why it's ok") else pure (IgnoreRule check target reason) --- | parse and validate an ignore file against the known check names parseIgnoreFile :: [Text] -> ByteString -> Either String IgnoreFile parseIgnoreFile known raw = do f@(IgnoreFile rules) <- eitherDecodeStrict raw@@ -73,24 +58,17 @@ bad -> Left ("unknown checks in ignore file: " <> T.unpack (T.intercalate ", " bad)) data Applied = Applied- { -- | findings that still count- apKept :: [Finding],- -- | findings an ignore matched+ { apKept :: [Finding], apIgnored :: [Finding],- -- | ignore file entries that matched nothing. kept apart from apProblems- -- because with several releases, an entry is only unused if it matched- -- nothing in any of them+ -- | kept apart from apProblems because with several releases, an entry is+ -- only unused if it matched nothing in any of them apUnused :: [Finding], -- | unknown check names in anda.hcl labels apProblems :: [Finding] } --- | split findings into kept and ignored. known is every check name, ran is--- the ones that ran this time (an entry for a check that didn't run isn't--- unused, just idle). labels maps a monorepo path to the check names its--- anda.hcl ignores applyIgnores :: [Text] -> [Text] -> IgnoreFile -> Map FilePath [Text] -> [Finding] -> Applied-applyIgnores known ran (IgnoreFile rules) labels findings =+applyIgnores knownChecks checksRun (IgnoreFile rules) ignoresByPath findings = Applied { apKept = kept, apIgnored = ignored,@@ -101,7 +79,7 @@ (ignored, kept) = partition isIgnored findings isIgnored f = byLabel f || any (`matches` f) rules byLabel f = case fPath f of- Just path -> fCheck f `elem` Map.findWithDefault [] (T.unpack path) labels+ Just path -> fCheck f `elem` Map.findWithDefault [] (T.unpack path) ignoresByPath Nothing -> False matches r f = irCheck r == fCheck f && case irTarget r of@@ -117,7 +95,8 @@ fReleases = [] } | r <- rules,- irCheck r `elem` ran,+ -- an entry for a check that didn't run is idle, not unused+ irCheck r `elem` checksRun, not (any (matches r) findings) ] badLabels =@@ -129,9 +108,9 @@ fMessage = "oglyad_ignore in anda.hcl names unknown check " <> c, fReleases = [] }- | (path, cs) <- Map.toList labels,+ | (path, cs) <- Map.toList ignoresByPath, c <- cs,- c `notElem` known+ c `notElem` knownChecks ] describe = \case ByPath p -> p
src/Oglyad/Process.hs view
@@ -1,5 +1,3 @@--- | running external tools (rpmspec, git) without ever throwing, hanging, or--- leaking secrets module Oglyad.Process ( run, runCapture,@@ -16,46 +14,43 @@ import Data.Text.Encoding qualified as TE import System.Environment (getEnvironment) import System.Exit (ExitCode (..))-import System.Process.Typed (nullStream, proc, readProcess, setEnv, setStdin, setWorkingDir)+import System.Process.Typed (ProcessConfig, nullStream, proc, readProcess, setEnv, setStdin, setWorkingDir) import System.Timeout (timeout) --- | run a command and return its stdout. a missing executable, a non-zero--- exit, or a timeout becomes a Left, so callers can report it as a finding--- instead of crashing the whole run.------ rpmspec runs shell code from specs (%(...) and %{lua:...}), so children get--- an allowlisted environment: GITHUB_TOKEN and anything else secret in ours--- never reaches them. stdin is closed so nothing can wait on a prompt+-- | rpmspec runs shell code from specs (%(...) and %{lua:...}), so children get+-- an allowlisted environment and GITHUB_TOKEN never reaches them. stdin is+-- closed so nothing can wait on a prompt run :: FilePath -> [String] -> IO (Either Text Text) run cmd args = do env <- childEnv <$> getEnvironment- r <- try (timeout (5 * 60 * 1_000_000) (readProcess (setEnv env (setStdin nullStream (proc cmd args)))))- pure $ case r of- Left (e :: IOException) -> Left (T.pack cmd <> ": " <> T.pack (show e))- Right Nothing -> Left (T.pack cmd <> " timed out after 5 minutes")- Right (Just (ExitSuccess, out, _)) -> Right (decode out)- Right (Just (ExitFailure _, _, err)) -> Left (T.strip (decode err))- where- decode = TE.decodeUtf8Lenient . LBS.toStrict+ result <- runConfined cmd (setEnv env (proc cmd args))+ pure $+ result >>= \case+ (ExitSuccess, out, _) -> Right out+ (ExitFailure _, _, err) -> Left (T.strip err) --- | like 'run', but in a working directory, with extra variables on top of--- the allowlist, and handing back the exit code, stdout and stderr even when--- the command succeeded. only for tools that need something the allowlist--- leaves out, like anda needing GITHUB_TOKEN+-- | for tools that need something the allowlist leaves out, like anda needing+-- GITHUB_TOKEN runCapture :: FilePath -> [(String, String)] -> FilePath -> [String] -> IO (Either Text (ExitCode, Text, Text)) runCapture dir extra cmd args = do env <- (extra <>) . filter ((`notElem` map fst extra) . fst) . childEnv <$> getEnvironment- r <- try (timeout (5 * 60 * 1_000_000) (readProcess (setWorkingDir dir (setEnv env (setStdin nullStream (proc cmd args))))))+ runConfined cmd (setWorkingDir dir (setEnv env (proc cmd args)))++timeoutMinutes :: Int+timeoutMinutes = 5++runConfined :: FilePath -> ProcessConfig stdin () () -> IO (Either Text (ExitCode, Text, Text))+runConfined cmd config = do+ r <- try (timeout (timeoutMinutes * 60 * 1_000_000) (readProcess (setStdin nullStream config))) pure $ case r of Left (e :: IOException) -> Left (T.pack cmd <> ": " <> T.pack (show e))- Right Nothing -> Left (T.pack cmd <> " timed out after 5 minutes")+ Right Nothing -> Left (T.pack cmd <> " timed out after " <> T.pack (show timeoutMinutes) <> " minutes") Right (Just (code, out, err)) -> Right (code, decode out, decode err) where decode = TE.decodeUtf8Lenient . LBS.toStrict --- | the environment children get: only what rpm and git need to find their--- config and behave, plus GIT_TERMINAL_PROMPT=0 so a git remote asking for--- credentials fails fast instead of hanging+-- | GIT_TERMINAL_PROMPT=0 makes a git remote asking for credentials fail fast+-- instead of hanging childEnv :: [(String, String)] -> [(String, String)] childEnv parent = ("GIT_TERMINAL_PROMPT", "0") : filter (allowed . fst) parent where
src/Oglyad/Regex.hs view
@@ -1,19 +1,15 @@--- | a regex engine for the dialect of Rust's regex crate, which andax's--- find(), find_all() and sub() use (andax/fns/kokoro.rs in FyraLabs/anda).+-- | the dialect of Rust's regex crate that andax's find(), find_all() and+-- sub() use (andax/fns/kokoro.rs in FyraLabs/anda). ----- it works the way Rust's BoundedBacktracker does: compile to a Thompson--- NFA shaped like regex-automata's compiler builds it, then backtrack with a--- visited set of (state, position) pairs. the visited set keeps it linear--- time, and it's also what decides which captures win when a repeated group--- can match empty, so mirroring the NFA shape gets the same answers as Rust.--- it was checked against the real crate on every regex in terrapkg/packages--- and ~300k fuzzed ones, with identical matches and captures.+-- it mirrors Rust's BoundedBacktracker: a Thompson NFA shaped like+-- regex-automata builds it, backtracked with a visited set of (state,+-- position) pairs. the visited set also decides which captures win when a+-- repeated group can match empty, so the NFA shape has to match Rust's to+-- get the same answers. it was checked against the real crate on every regex+-- in terrapkg/packages and ~300k fuzzed ones. ----- supported: literals and escapes, ., classes with ranges, negation and--- [:posix:] names, \d \w \s and their negations, \b \B \A \z, ^ $, groups--- ((...), (?:...), (?P<name>...), (?<name>...)), alternation, * + ? {n}--- {n,} {n,m} and their lazy forms, and the i, m and s flags. anything else--- is a compile error, which callers treat as "can't tell"+-- anything the parser doesn't handle, like \p{...}, nested classes or class+-- operations, is a compile error, which callers treat as "can't tell" module Oglyad.Regex ( Regex, compile,@@ -35,19 +31,21 @@ import Data.Vector.Unboxed.Mutable qualified as MVU import Text.Read (readMaybe) +type Multiline = Bool++type WantBoundary = Bool++type Greedy = Bool+ data Node- = -- | one character- One (Char -> Bool)- | -- | ^ and $, True for multi-line- LineStart Bool- | LineEnd Bool- | -- | \b, False for \B- WordBoundary Bool+ = One (Char -> Bool)+ | LineStart Multiline+ | LineEnd Multiline+ | WordBoundary WantBoundary | Group (Maybe Int) Node | Alt [Node] | Seq [Node]- | -- | min, max, greedy- Repeat Int (Maybe Int) Bool Node+ | Repeat Int (Maybe Int) Greedy Node data Regex = Regex { reProg :: Prog,@@ -57,18 +55,19 @@ data Flags = Flags {fI, fM, fS :: Bool} --- * parsing---- | parser state: input left, groups numbered so far, names seen-data St = St {stIn :: String, stGroups :: Int, stNames :: [(Text, Int)]}+data ParseState = ParseState+ { stRest :: String,+ stGroupCount :: Int,+ stGroupNames :: [(Text, Int)]+ } -type Parse a = St -> Either Text (a, St)+type Parse a = ParseState -> Either Text (a, ParseState) compile :: Text -> Either Text Regex compile src = do- (node, st) <- alternation (Flags False False False) (St (T.unpack src) 0 [])- case stIn st of- [] -> Right (Regex (toProg (stGroups st) node) (stGroups st) (stNames st))+ (node, st) <- alternation (Flags False False False) (ParseState (T.unpack src) 0 [])+ case stRest st of+ [] -> Right (Regex (toProg (stGroupCount st) node) (stGroupCount st) (stGroupNames st)) c : _ -> Left ("unexpected " <> T.singleton c) -- | a|b|c, up to a ')' or the end. (?flags) partway through applies to the@@ -78,23 +77,22 @@ where go fl st acc = do ((node, fl'), st') <- sequence' fl st []- case stIn st' of- '|' : rest -> go fl' st' {stIn = rest} (node : acc)+ case stRest st' of+ '|' : rest -> go fl' st' {stRest = rest} (node : acc) _ -> Right (alt (reverse (node : acc)), st') alt [n] = n alt ns = Alt ns -sequence' :: Flags -> St -> [Node] -> Either Text ((Node, Flags), St)-sequence' fl st acc = case stIn st of+sequence' :: Flags -> ParseState -> [Node] -> Either Text ((Node, Flags), ParseState)+sequence' fl st acc = case stRest st of [] -> done '|' : _ -> done ')' : _ -> done- -- a bare flag group changes the flags for what follows '(' : '?' : rest | (fs, ')' : rest') <- span (`elem` ("imsxUR-" :: String)) rest, not (null fs) -> do fl' <- setFlags fl fs- sequence' fl' st {stIn = rest'} acc+ sequence' fl' st {stRest = rest'} acc _ -> do (atom, st1) <- atomP fl st (node, st2) <- quantifier atom st1@@ -114,7 +112,7 @@ _ -> Left ("unsupported flag " <> T.singleton c) quantifier :: Node -> Parse Node-quantifier atom st = case stIn st of+quantifier atom st = case stRest st of '*' : rest -> lazy 0 Nothing rest '+' : rest -> lazy 1 Nothing rest '?' : rest -> lazy 0 (Just 1) rest@@ -133,42 +131,42 @@ (Just lo, Just hi) | maybe True (>= lo) hi -> lazy lo hi rest _ -> Left "bad repetition" lazy lo hi = \case- '?' : rest -> Right (Repeat lo hi False atom, st {stIn = rest})- rest -> Right (Repeat lo hi True atom, st {stIn = rest})+ '?' : rest -> Right (Repeat lo hi False atom, st {stRest = rest})+ rest -> Right (Repeat lo hi True atom, st {stRest = rest}) atomP :: Flags -> Parse Node-atomP fl st = case stIn st of- '(' : '?' : ':' : rest -> group Nothing st {stIn = rest}+atomP fl st = case stRest st of+ '(' : '?' : ':' : rest -> group Nothing st {stRest = rest} '(' : '?' : 'P' : '<' : rest -> named rest '(' : '?' : '<' : rest -> named rest '(' : '?' : _ -> Left "unsupported group" '(' : rest ->- let n = stGroups st + 1- in group (Just n) st {stIn = rest, stGroups = n}+ let n = stGroupCount st + 1+ in group (Just n) st {stRest = rest, stGroupCount = n} '[' : rest -> do- (p, rest') <- classP fl rest- Right (One p, st {stIn = rest'})- '.' : rest -> Right (One (\c -> fS fl || c /= '\n'), st {stIn = rest})- '^' : rest -> Right (LineStart (fM fl), st {stIn = rest})- '$' : rest -> Right (LineEnd (fM fl), st {stIn = rest})+ (p, rest') <- bracketClass fl rest+ Right (One p, st {stRest = rest'})+ '.' : rest -> Right (One (\c -> fS fl || c /= '\n'), st {stRest = rest})+ '^' : rest -> Right (LineStart (fM fl), st {stRest = rest})+ '$' : rest -> Right (LineEnd (fM fl), st {stRest = rest}) '\\' : rest -> do (node, rest') <- escape fl rest- Right (node, st {stIn = rest'})+ Right (node, st {stRest = rest'}) c : rest | c `elem` ("*+?{" :: String) -> Left "repetition with nothing to repeat"- | otherwise -> Right (One (charEq fl c), st {stIn = rest})+ | otherwise -> Right (One (charEq fl c), st {stRest = rest}) [] -> Left "unexpected end" where group idx s = do (inner, s') <- alternation fl s- case stIn s' of- ')' : rest -> Right (Group idx inner, s' {stIn = rest})+ case stRest s' of+ ')' : rest -> Right (Group idx inner, s' {stRest = rest}) _ -> Left "unclosed group" named rest = case span (\c -> isAlphaNum c || c == '_') rest of (name, '>' : rest') | not (null name) ->- let n = stGroups st + 1- in group (Just n) st {stIn = rest', stGroups = n, stNames = (T.pack name, n) : stNames st}+ let n = stGroupCount st + 1+ in group (Just n) st {stRest = rest', stGroupCount = n, stGroupNames = (T.pack name, n) : stGroupNames st} _ -> Left "bad group name" charEq :: Flags -> Char -> Char -> Bool@@ -176,7 +174,6 @@ | fI fl = \x -> toLower x == toLower c || toUpper x == toUpper c | otherwise = (== c) --- | an escape outside a class escape :: Flags -> String -> Either Text (Node, String) escape fl = \case 'b' : rest -> Right (WordBoundary True, rest)@@ -187,11 +184,10 @@ (p, rest) <- classEscape fl s Right (One p, rest) --- | an escape that stands for one character or a class of them classEscape :: Flags -> String -> Either Text (Char -> Bool, String) classEscape fl = \case- 'd' : rest -> Right (isDigit', rest)- 'D' : rest -> Right (not . isDigit', rest)+ 'd' : rest -> Right (isUnicodeDigit, rest)+ 'D' : rest -> Right (not . isUnicodeDigit, rest) 'w' : rest -> Right (isWord, rest) 'W' : rest -> Right (not . isWord, rest) 's' : rest -> Right (isSpace, rest)@@ -208,16 +204,14 @@ where lit c rest = Right (charEq fl c, rest) --- | Rust's \d is Unicode-aware, and isDigit is ASCII-only-isDigit' :: Char -> Bool-isDigit' c = generalCategory c == DecimalNumber+isUnicodeDigit :: Char -> Bool+isUnicodeDigit c = generalCategory c == DecimalNumber isWord :: Char -> Bool isWord c = isAlphaNum c || c == '_' --- | the inside of [...], given what's after the '['-classP :: Flags -> String -> Either Text (Char -> Bool, String)-classP fl s0 = do+bracketClass :: Flags -> String -> Either Text (Char -> Bool, String)+bracketClass fl s0 = do let (negated, s1) = case s0 of '^' : r -> (True, r) r -> (False, r)@@ -239,26 +233,24 @@ '-' : '-' : _ -> Left "class operations aren't supported" '~' : '~' : _ -> Left "class operations aren't supported" s -> do- (lo, rest) <- single s+ (lo, rest) <- classItem s case (lo, rest) of- (Left c, '-' : rest') | take 1 rest' /= "]" -> do- (hi, rest'') <- single rest'+ (ItemChar c, '-' : rest') | take 1 rest' /= "]" -> do+ (hi, rest'') <- classItem rest' case hi of- Left d | c <= d -> go (range c d : acc) rest''+ ItemChar d | c <= d -> go (range c d : acc) rest'' _ -> Left "bad class range"- (Left c, _) -> go (charEq fl c : acc) rest- (Right p, _) -> go (p : acc) rest- -- one class item: Left for a plain character (which can start a range),- -- Right for an escape like \d- single = \case+ (ItemChar c, _) -> go (charEq fl c : acc) rest+ (ItemSet p, _) -> go (p : acc) rest+ classItem = \case '\\' : rest -> case rest of- c : rest' | isPunctuation c || isSymbol c || c == ' ' -> Right (Left c, rest')- 'n' : rest' -> Right (Left '\n', rest')- 't' : rest' -> Right (Left '\t', rest')+ c : rest' | isPunctuation c || isSymbol c || c == ' ' -> Right (ItemChar c, rest')+ 'n' : rest' -> Right (ItemChar '\n', rest')+ 't' : rest' -> Right (ItemChar '\t', rest') _ -> do (p, rest') <- classEscape fl rest- Right (Right p, rest')- c : rest -> Right (Left c, rest)+ Right (ItemSet p, rest')+ c : rest -> Right (ItemChar c, rest) [] -> Left "unclosed class" range lo hi | fI fl = \c -> any (\x -> lo <= x && x <= hi) [c, toLower c, toUpper c]@@ -275,7 +267,7 @@ _ -> Left "unknown posix class" isAlphaNumAscii c = isDigit c || isAsciiLower c || isAsciiUpper c --- * compiling to an NFA+data ClassItem = ItemChar Char | ItemSet (Char -> Bool) data Look = AtLineStart Bool | AtLineEnd Bool | AtWordBoundary Bool @@ -283,41 +275,37 @@ = SChar (Char -> Bool) !Int | -- | alternatives in preference order SUnion [Int]- | -- | record the position in a capture slot- SCapture !Int !Int+ | SCapture !Int !Int | SLook Look !Int | SMatch data Prog = Prog { progStates :: V.Vector State, progStart :: Int,- -- | two per group, group 0 being the whole match+ -- | a start and an end per group, group 0 being the whole match progSlots :: Int } --- | NFA under construction: the next free id and the states so far-type Build = (Int, IntMap State)+data Build = Build {nextId :: !Int, builtStates :: IntMap State} new :: State -> Build -> (Int, Build)-new st (n, m) = (n, (n + 1, IM.insert n st m))+new st b@Build {..} = (nextId, b {nextId = nextId + 1, builtStates = IM.insert nextId st builtStates}) --- | reserve a state to fill in later, for loops reserve :: Build -> (Int, Build)-reserve (n, m) = (n, (n + 1, m))+reserve b@Build {..} = (nextId, b {nextId = nextId + 1}) fill :: Int -> State -> Build -> Build-fill i st (n, m) = (n, IM.insert i st m)+fill i st b@Build {..} = b {builtStates = IM.insert i st builtStates} toProg :: Int -> Node -> Prog toProg groups node =- let (matchId, b1) = new SMatch (0, IM.empty)+ let (matchId, b1) = new SMatch (Build 0 IM.empty) (end0, b2) = new (SCapture 1 matchId) b1 (body, b3) = nfa node end0 b2- (start, (n, states)) = new (SCapture 0 body) b3+ (start, Build {nextId = n, builtStates = states}) = new (SCapture 0 body) b3 in Prog (V.generate n (states IM.!)) start (2 * (groups + 1)) --- | compile a node that continues to `next`, returning its start state.--- the repetition shapes follow regex-automata's thompson compiler+-- | the repetition shapes follow regex-automata's thompson compiler -- (c_at_least, c_bounded, c_zero_or_one), since the visited set makes the -- shape observable nfa :: Node -> Int -> Build -> (Int, Build)@@ -345,7 +333,7 @@ | otherwise -> -- the optional copies nest: once one is skipped, the rest are too let (opt, b1) = foldr (\_ (nx, bb) -> let (x, bb') = nfa inner nx bb in new (SUnion (order x next)) bb') (next, b) [1 .. m - lo]- in copies' lo opt b1+ in copies lo opt b1 Nothing | lo == 0 && minLen inner > 0 -> let (u, b1) = reserve b@@ -359,18 +347,14 @@ b3 = fill plus (SUnion (order x next)) b2 in new (SUnion (order x next)) b3 | otherwise ->- -- n-1 copies, then one more that loops- let (u, b1) = reserve b- (x, b2) = nfa inner u b1- b3 = fill u (SUnion (order x next)) b2- in copies' (lo - 1) x b3+ let (loop, b1) = reserve b+ (lastCopy, b2) = nfa inner loop b1+ b3 = fill loop (SUnion (order lastCopy next)) b2+ in copies (lo - 1) lastCopy b3 where order x nx = if greedy then [x, nx] else [nx, x]- copies n nx bb = copies' n nx bb- copies' n nx bb = foldr (\_ (s, bb') -> nfa inner s bb') (nx, bb) [1 .. n]+ copies n nx bb = foldr (\_ (s, bb') -> nfa inner s bb') (nx, bb) [1 .. n] --- | the shortest match a node can make, to tell whether x* can loop on--- nothing minLen :: Node -> Int minLen = \case One _ -> 1@@ -382,18 +366,14 @@ Alt ns -> if null ns then 0 else minimum (map minLen ns) Repeat lo _ _ n -> lo * minLen n --- * matching---- | positions the visited set can track before we give up. Rust switches to--- another engine past its limit. we only ever see version strings, so a--- refusal here is fine+-- | Rust switches to another engine past its limit. we only ever see version+-- strings, so refusing is fine visitedLimit :: Int visitedLimit = 20000000 data Frame = Step !Int !Int | Restore !Int !Int --- | the first match starting at or after a position, as capture slots.--- the visited set lives for the whole search, like Rust's: a (state,+-- | the visited set lives for the whole search, like Rust's. a (state, -- position) that failed from one start fails from every later one too searchFrom :: Regex -> VU.Vector Char -> Int -> Either Text (Maybe (VU.Vector Int)) searchFrom re input from@@ -444,11 +424,10 @@ AtWordBoundary want -> (wordAt (at - 1) /= wordAt at) == want wordAt i = i >= 0 && i < len && isWord (input VU.! i) --- | (start, end, slots) of a match-type Match = (Int, Int, VU.Vector Int)+data Match = Match {matchStart, matchEnd :: Int, matchSlots :: VU.Vector Int} toMatch :: VU.Vector Int -> Match-toMatch slots = (slots VU.! 0, slots VU.! 1, slots)+toMatch slots = Match (slots VU.! 0) (slots VU.! 1) slots -- | every non-overlapping match, like Rust's captures_iter. an empty match -- right where the previous one ended is skipped@@ -460,15 +439,14 @@ | otherwise = searchFrom re input from >>= \case Nothing -> Right []- Just slots -> case toMatch slots of- (s, e, _)- | s == e && Just e == lastEnd -> go (s + 1) lastEnd- m@(s, e, _) -> (m :) <$> go (if s == e then e + 1 else e) (Just e)+ Just slots+ | s == e && Just e == lastEnd -> go (s + 1) lastEnd+ | otherwise -> (m :) <$> go (if s == e then e + 1 else e) (Just e)+ where+ m@Match {matchStart = s, matchEnd = e} = toMatch slots --- | group texts for one match, group 0 first. Nothing for groups that--- didn't take part groupsOf :: Regex -> VU.Vector Char -> Match -> [Maybe Text]-groupsOf re input (_, _, slots) =+groupsOf re input Match {matchSlots = slots} = [ if a >= 0 && b >= 0 then Just (T.pack (VU.toList (VU.slice a (b - a) input))) else Nothing | g <- [0 .. reGroups re], let a = slots VU.! (2 * g)@@ -478,8 +456,6 @@ chars :: Text -> VU.Vector Char chars = VU.fromList . T.unpack --- | the first match, like Rust's Regex::captures. Left when the text is--- too long for the engine captures :: Regex -> Text -> Either Text (Maybe [Maybe Text]) captures re t = fmap (groupsOf re input . toMatch) <$> searchFrom re input 0 where@@ -499,7 +475,7 @@ input = chars t slice a b = T.pack (VU.toList (VU.slice a (b - a) input)) go pos [] = [slice pos (VU.length input)]- go pos (m@(s, e, _) : ms) = slice pos s : expand (groupsOf re input m) : go e ms+ go pos (m@Match {matchStart = s, matchEnd = e} : ms) = slice pos s : expand (groupsOf re input m) : go e ms expand groups = T.pack (expandS groups (T.unpack rep)) expandS groups = \case [] -> []
src/Oglyad/Release.hs view
@@ -1,24 +1,33 @@--- | which Fedora releases to check, and everything that follows from one and--- the distro being checked: the repo, the %{dist} tag, the monorepo branch--- and the Fedora repos. also merges findings from several releases into one--- report module Oglyad.Release- ( Distro (..),+ ( Base (..),+ baseName,+ baseRepoOwners,+ Distro (..), terra,+ terraEl, ultramarine,+ tuatara, distros, lookupDistro,+ remoteRepoName, Release (..),- isRawhide,+ isRolling,+ tumbleweed,+ el,+ elReleasesFrom, releaseLabel, releaseDist, releaseBranch,+ releaseMacros,+ releaseSubrepos,+ releaseCacheTag, repoName, defaultRepoUrl, repoUrlFor,- fedoraReposFor,+ baseReposFor, parseReleaseArg, ReleaseArg (..),+ releaseArgBase, bodhiReleasesUrl, parseBodhiReleases, mergeFindings,@@ -38,114 +47,263 @@ import Oglyad.Types (Finding (..)) import Text.Read (readMaybe) --- | an anda monorepo oglyad knows how to check, and how it spells things per--- release+data Base = Fedora | OpenSUSE | EL+ deriving stock (Eq, Show)++baseName :: Base -> Text+baseName = \case+ Fedora -> "Fedora"+ OpenSUSE -> "openSUSE"+ EL -> "EL"++-- | who ships the packages in 'baseReposFor'+baseRepoOwners :: Base -> Text+baseRepoOwners = \case+ EL -> "AlmaLinux or EPEL"+ b -> baseName b+ data Distro = Distro- { -- | what --distro takes, e.g. "terra"- distroName :: Text,- -- | the directory under the monorepo root holding every package, e.g.- -- "anda" for terrapkg/packages+ { distroName :: Text,+ -- | releases default to the first one's+ distroBases :: [Base], distroRoot :: FilePath,- -- | the repo's name, with the release name appended: terra for terra43+ distroMarker :: FilePath,+ -- | the monorepo's GitHub repo name, which tells distros with the same+ -- marker apart+ distroUpstream :: Text, distroRepo :: Text,- -- | release branch prefix, f for f43 distroBranch :: Text,- -- | %{dist} prefix, .fc for .fc43- distroDist :: Text+ distroDist :: Text,+ distroRepoUrl :: Text,+ distroSubrepos :: Bool,+ distroSkippedChecks :: [Text] } deriving stock (Eq, Show) --- | terrapkg/packages+-- | its el10 branch is still around until Terra EL finishes moving to+-- 'terraEl' terra :: Distro-terra = Distro "terra" "anda" "terra" "f" ".fc"+terra =+ Distro+ { distroName = "terra",+ distroBases = [Fedora, EL],+ distroRoot = "anda",+ distroMarker = "anda",+ distroUpstream = "packages",+ distroRepo = "terra",+ distroBranch = "f",+ distroDist = ".fc",+ distroRepoUrl = defaultRepoUrl,+ distroSubrepos = True,+ distroSkippedChecks = []+ } --- | Ultramarine-Linux/packages. builds land in um43, umrawhide, with--- .um43 and .umrawhide dist tags+terraEl :: Distro+terraEl =+ Distro+ { distroName = "terra-el",+ distroBases = [EL],+ distroRoot = "anda",+ distroMarker = "anda",+ distroUpstream = "packages-el",+ distroRepo = "terra",+ distroBranch = "",+ distroDist = "",+ distroRepoUrl = defaultRepoUrl,+ distroSubrepos = False,+ distroSkippedChecks = []+ }+ ultramarine :: Distro-ultramarine = Distro "ultramarine" "ultramarine" "um" "um" ".um"+ultramarine =+ Distro+ { distroName = "ultramarine",+ distroBases = [Fedora],+ distroRoot = "ultramarine",+ distroMarker = "ultramarine",+ distroUpstream = "packages",+ distroRepo = "um",+ distroBranch = "um",+ distroDist = ".um",+ distroRepoUrl = defaultRepoUrl,+ distroSubrepos = True,+ distroSkippedChecks = ["no-autoupdate"]+ } +tuatara :: Distro+tuatara =+ Distro+ { distroName = "tuatara",+ distroBases = [OpenSUSE],+ distroRoot = "",+ distroMarker = "tuatara",+ distroUpstream = "tuatara",+ distroRepo = "tuatara-",+ distroBranch = "",+ distroDist = "",+ distroRepoUrl = "https://repos.fyralabs.com/{repo}",+ distroSubrepos = False,+ distroSkippedChecks = []+ }+ distros :: [Distro]-distros = [terra, ultramarine]+distros = [terra, terraEl, ultramarine, tuatara] lookupDistro :: String -> Either String Distro lookupDistro raw = case [d | d <- distros, distroName d == T.pack raw] of d : _ -> Right d [] -> Left ("unknown distro " <> show raw <> ", expected one of: " <> T.unpack (T.intercalate ", " (map distroName distros))) --- | a Fedora release Terra builds for. rawhide keeps its number too, since--- %fedora still needs one when evaluating specs+remoteRepoName :: Text -> Maybe Text+remoteRepoName url = case T.split (`elem` ['/', ':']) (T.dropWhileEnd (== '/') (T.strip url)) of+ parts@(_ : _ : _) | name <- dropGit (last parts), not (T.null name) -> Just name+ _ -> Nothing+ where+ dropGit n = maybe n id (T.stripSuffix ".git" n)++-- | a base distro release a monorepo builds for. rawhide keeps its number+-- too, since %fedora still needs one when evaluating specs data Release = Release- { -- | "43", or "rawhide"+ { relBase :: Base, relName :: Text,- -- | the number %fedora expands to+ -- | what %fedora, %suse_version or %rhel expands to relVersion :: Int } deriving stock (Eq, Show) -isRawhide :: Release -> Bool-isRawhide r = relName r == "rawhide"+isRolling :: Release -> Bool+isRolling r = relName r `elem` ["rawhide", "tumbleweed"] --- | how releases are spelled in output: f43, rawhide. the same for every--- distro, since they're all built on Fedora's releases+tumbleweed :: Release+tumbleweed = Release OpenSUSE "tumbleweed" 1699++el :: Int -> Release+el n = Release EL ("el" <> T.pack (show n)) n++elReleasesFrom :: [Text] -> [Release]+elReleasesFrom branches =+ sortOn+ relVersion+ [ el n+ | b <- nub branches,+ Just digits <- [T.stripPrefix "el" b],+ not (T.null digits),+ T.all isDigit digits,+ Just n <- [readMaybe (T.unpack digits)]+ ]+ releaseLabel :: Release -> Text-releaseLabel r = if isRawhide r then "rawhide" else "f" <> relName r+releaseLabel r+ | relBase r == Fedora && not (isRolling r) = "f" <> relName r+ | otherwise = relName r -- | rawhide builds use .fcrawhide, not .fc46. terrapkg/packages' bootstrap -- workflow rewrites macros.dist to do it, and Ultramarine does the same releaseDist :: Distro -> Release -> Text-releaseDist d r = distroDist d <> relName r+releaseDist d r = case relBase r of+ Fedora -> distroDist d <> relName r+ OpenSUSE -> ""+ EL -> "." <> relName r --- | the monorepo branch for a release: f43, frawhide, um43+-- | %fedora comes from the host otherwise, so -r 43 on an F44 box would+-- take the wrong side of every %if 0%{?fedora} >= 44+releaseMacros :: Distro -> Release -> [(Text, Maybe Text)]+releaseMacros d r = case relBase r of+ Fedora -> [("dist", Just (releaseDist d r)), ("fedora", Just version)]+ OpenSUSE ->+ [ ("dist", Nothing),+ ("fedora", Nothing),+ ("suse_version", Just version),+ ("is_opensuse", Just "1")+ ]+ EL ->+ [ ("dist", Just (releaseDist d r)),+ ("fedora", Nothing),+ ("rhel", Just version),+ (relName r, Just "1"),+ ("epel", Just version)+ ]+ where+ version = T.pack (show (relVersion r))++releaseCacheTag :: Distro -> Release -> Text+releaseCacheTag d r = "-" <> distroName d <> "-" <> relName r+ releaseBranch :: Distro -> Release -> Text-releaseBranch d r = distroBranch d <> relName r+releaseBranch d r = case relBase r of+ Fedora -> distroBranch d <> relName r+ _ -> relName r --- | the distro's repo for a release, e.g. terra43 or umrawhide. subrepos--- add a suffix: terra43-extras+-- | Terra's EL builds only ever upload to terrael10, whatever subrepo label+-- a package has+releaseSubrepos :: Distro -> Release -> Bool+releaseSubrepos d r = distroSubrepos d && relBase r == Fedora+ repoName :: Distro -> Release -> Text repoName d r = distroRepo d <> relName r --- | where Fyra Labs' repos are, the same metalink dnf uses. {repo} becomes--- a 'repoName', with any subrepo suffix+-- | the same metalink dnf uses defaultRepoUrl :: Text defaultRepoUrl = "https://tetsudou.fyralabs.com/metalink?repo={repo}&arch=x86_64" --- | a repo url template for one release, with {release} filled in and--- {repo} left for 'Oglyad.Types.repoUrl'+-- | {repo} is left for 'Oglyad.Types.repoUrl' to fill in repoUrlFor :: Text -> Release -> Text repoUrlFor url r = T.replace "{release}" (relName r) url --- | Fedora's own repos, for the overlap check. rawhide has no updates repo-fedoraReposFor :: Release -> [Text]-fedoraReposFor r- | isRawhide r = ["https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os"]- | otherwise =- [ "https://dl.fedoraproject.org/pub/fedora/linux/releases/" <> relName r <> "/Everything/x86_64/os",- "https://dl.fedoraproject.org/pub/fedora/linux/updates/" <> relName r <> "/Everything/x86_64"- ]+-- | EL builds run on AlmaLinux with EPEL enabled, so both count+baseReposFor :: Release -> [Text]+baseReposFor r = case relBase r of+ OpenSUSE ->+ [ "https://download.opensuse.org/tumbleweed/repo/oss",+ "https://download.opensuse.org/update/tumbleweed"+ ]+ EL ->+ [ "https://repo.almalinux.org/almalinux/" <> major <> "/" <> repo <> "/x86_64/os"+ | repo <- ["BaseOS", "AppStream", "CRB", "extras"]+ ]+ <> ["https://dl.fedoraproject.org/pub/epel/" <> major <> "/Everything/x86_64"]+ Fedora -> fedoraRepos+ where+ major = T.pack (show (relVersion r))+ fedoraRepos+ | isRolling r = ["https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os"]+ | otherwise =+ [ "https://dl.fedoraproject.org/pub/fedora/linux/releases/" <> relName r <> "/Everything/x86_64/os",+ "https://dl.fedoraproject.org/pub/fedora/linux/updates/" <> relName r <> "/Everything/x86_64"+ ] --- | what --release accepts, before we know rawhide's number-data ReleaseArg = NumberedArg Int | RawhideArg+-- | --release, before we know the distro or rawhide's number+data ReleaseArg = NumberedArg Int | RawhideArg | TumbleweedArg | ElArg Int deriving stock (Eq, Show) --- | 43, f43, rawhide or frawhide+releaseArgBase :: ReleaseArg -> Base+releaseArgBase = \case+ NumberedArg _ -> Fedora+ RawhideArg -> Fedora+ TumbleweedArg -> OpenSUSE+ ElArg _ -> EL+ parseReleaseArg :: String -> Either String ReleaseArg-parseReleaseArg raw = case dropF raw of- "rawhide" -> Right RawhideArg- n- | not (null n), all isDigit n, Just v <- readMaybe n -> Right (NumberedArg v)- _ -> Left ("unknown release " <> show raw <> ", expected a number like 43, f43, or rawhide")+parseReleaseArg raw = case raw of+ "tumbleweed" -> Right TumbleweedArg+ "tw" -> Right TumbleweedArg+ 'e' : 'l' : n | Just v <- number n -> Right (ElArg v)+ _ -> case dropF raw of+ "rawhide" -> Right RawhideArg+ n | Just v <- number n -> Right (NumberedArg v)+ _ -> Left ("unknown release " <> show raw <> ", expected a number like 43, f43, rawhide, tumbleweed, or el10") where dropF ('f' : rest) = rest dropF s = s+ number n+ | not (null n), all isDigit n = readMaybe n+ | otherwise = Nothing --- | every Fedora release Bodhi hasn't archived. that's the current ones, a--- branched one before its release, and rawhide+-- | unarchived releases are the current ones, a branched one before its+-- release, and rawhide bodhiReleasesUrl :: Text bodhiReleasesUrl = "https://bodhi.fedoraproject.org/releases/?exclude_archived=true&rows_per_page=100" --- | the maintained Fedora releases in a Bodhi /releases/ response, oldest--- first with rawhide last. ELN, containers and flatpaks are skipped: only--- names like F43 count parseBodhiReleases :: LBS.ByteString -> Either Text [Release] parseBodhiReleases body = do v <- either (Left . ("bad json from Bodhi: " <>) . T.pack) Right (eitherDecode body)@@ -153,24 +311,25 @@ Object o | Just (Array rs) <- KM.lookup "releases" o -> Right (V.toList rs) _ -> Left "no releases in Bodhi's answer" let found =- [ Release (if branch == "rawhide" then "rawhide" else ver) n+ [ Release Fedora (if branch == "rawhide" then "rawhide" else ver) n | Object r <- rels, Just (String name) <- [KM.lookup "name" r],- Just ('F', digits) <- [T.uncons name],- not (T.null digits),- T.all isDigit digits,+ isFedoraRelease name, Just (String ver) <- [KM.lookup "version" r], Just n <- [readMaybe (T.unpack ver)], Just (String branch) <- [KM.lookup "branch" r] ] case found of [] -> Left "Bodhi listed no Fedora releases"- _ -> Right (sortOn (\r -> (isRawhide r, relVersion r)) found)+ _ -> Right (sortOn (\r -> (isRolling r, relVersion r)) found)+ where+ isFedoraRelease name = case T.uncons name of+ Just ('F', digits) -> not (T.null digits) && T.all isDigit digits+ _ -> False --- | one list of findings out of each release's. a finding that comes up the--- same way in several releases is shown once, with fReleases naming them.--- "the same way" ignores the dist tag, so build-lag's--- "spec is at 1.2-1.fc43" and "spec is at 1.2-1.fc44" merge into+-- | a finding that comes up the same way in several releases is shown once,+-- with fReleases naming them. "the same way" ignores the dist tag, so+-- build-lag's "spec is at 1.2-1.fc43" and "spec is at 1.2-1.fc44" merge into -- "spec is at 1.2-1%{dist}" mergeFindings :: Distro -> [(Release, [Finding])] -> [Finding] mergeFindings distro perRelease =@@ -180,7 +339,10 @@ ] where tagged = [(keyOf r f, (f, relName r)) | (r, fs) <- perRelease, f <- fs]- keyOf r f = (fCheck f, fPackage f, fPath f, fSeverity f, T.replace (releaseDist distro r) "%{dist}" (fMessage f))+ keyOf r f = (fCheck f, fPackage f, fPath f, fSeverity f, distless r (fMessage f))+ distless r msg = case releaseDist distro r of+ "" -> msg+ dist -> T.replace dist "%{dist}" msg -- members stay in the order the releases were given grouped = Map.fromListWith (flip (<>)) [(k, [x]) | (k, x) <- tagged] merged (_, _, _, _, generic) f rels = case nub rels of
src/Oglyad/Report.hs view
@@ -1,5 +1,3 @@--- | rendering findings as terminal text, json for scripts, or markdown for--- GitHub job summaries module Oglyad.Report ( Report (..), ReleaseInfo (..),@@ -26,26 +24,22 @@ data Format = Text | Json | Markdown deriving stock (Eq, Show) --- | everything a report needs besides the findings themselves data Report = Report { rVersion :: Text, rGenerated :: UTCTime,- -- | every release this run checked, in order rReleases :: [ReleaseInfo],- -- | "terra", "ultramarine" rDistro :: Text, rUpstream :: Bool, rFindings :: [Finding], rIgnored :: [Finding] } --- | what one release was checked against data ReleaseInfo = ReleaseInfo- { -- | "43", "rawhide", as findings' fReleases spell it+ { -- | "43", as fReleases spells it riName :: Text,- -- | "f43", "rawhide", for people+ -- | "f43", for people riLabel :: Text,- -- | the git ref the specs came from, Nothing for the working tree+ -- | Nothing for the working tree riRef :: Maybe Text, riCommit :: Maybe Text, riRepo :: Text@@ -59,8 +53,6 @@ where utf8 = LBS.fromStrict . TE.encodeUtf8 --- * text- renderText :: Severity -> Report -> Text renderText minSev r@Report {..} = T.unlines (map line (shown minSev rFindings) <> ["", summaryLine rFindings rIgnored])@@ -69,7 +61,7 @@ line f@Finding {..} = T.justifyLeft 6 ' ' (sevText fSeverity) <> T.justifyLeft 16 ' ' fCheck- <> label amb f+ <> packageLabel amb f <> releaseTag r f <> ": " <> fMessage@@ -81,15 +73,16 @@ count s = tshow (length (filter ((== s) . fSeverity) fs)) ignoredPart = if null ignored then "" else " (" <> tshow (length ignored) <> " ignored)" --- * json+-- | bump on any breaking change to field names or meanings, so scripts reading+-- the json can tell+schemaVersion :: Int+schemaVersion = 2 --- | a stable envelope. bump schema on any breaking change to field names or--- meanings, so scripts reading this can tell renderJson :: Report -> LBS.ByteString renderJson Report {..} = encode $ object- [ "schema" .= (2 :: Int),+ [ "schema" .= schemaVersion, "oglyad" .= rVersion, "generated" .= rGenerated, "distro" .= rDistro,@@ -127,10 +120,7 @@ "releases" .= fReleases ] --- * markdown---- | sized for a GitHub job summary: counts first, errors and warnings in--- full, info folded away+-- | sized for a GitHub job summary, so info is folded away renderMarkdown :: Severity -> Report -> Text renderMarkdown minSev r@Report {..} = T.unlines $@@ -168,35 +158,28 @@ let heading = title <> " (" <> tshow (length fs) <> ")" table = ["", "| package | check | message |", "|---|---|---|"]- <> [ "| " <> cell (label amb f <> releaseTag r f) <> " | `" <> fCheck f <> "` | " <> cell (fMessage f) <> " |"+ <> [ "| " <> escapeCell (packageLabel amb f <> releaseTag r f) <> " | `" <> fCheck f <> "` | " <> escapeCell (fMessage f) <> " |" | f <- fs ] in if folded then ["", "<details><summary>" <> heading <> "</summary>"] <> table <> ["", "</details>"] else ["", "## " <> heading] <> table- -- table cells can't contain raw pipes or newlines- cell = T.replace "|" "\\|" . T.replace "\n" " "---- * shared+ escapeCell = T.replace "|" "\\|" . T.replace "\n" " " shown :: Severity -> [Finding] -> [Finding] shown minSev = sortOn (\f -> (Down (fSeverity f), fCheck f, fPackage f)) . filter ((>= minSev) . fSeverity) --- | the package name, plus its directory when the name belongs to several-label :: Set.Set Text -> Finding -> Text-label ambiguous Finding {..} =+packageLabel :: Set.Set Text -> Finding -> Text+packageLabel ambiguous Finding {..} = fPackage <> maybe "" (\path -> if fPackage `Set.member` ambiguous then " (" <> path <> ")" else "") fPath --- | " [f43, f44]" for a finding that only some of the checked releases--- have. nothing when it's in all of them, or only one release was checked releaseTag :: Report -> Finding -> Text releaseTag Report {rReleases} Finding {fReleases} | length rReleases <= 1 = "" | all (`elem` fReleases) (map riName rReleases) = "" | otherwise = " [" <> T.intercalate ", " [riLabel ri | ri <- rReleases, riName ri `elem` fReleases] <> "]" --- | names that show up with more than one directory ambiguousNames :: [Finding] -> Set.Set Text ambiguousNames fs = Map.keysSet . Map.filter ((> 1) . Set.size) $
src/Oglyad/Source/Anda.hs view
@@ -1,32 +1,37 @@--- | running the real andax on update.rhai scripts oglyad can't follow on its--- own, like ones that scrape a page with get() and find().------ anda rewrites specs in place, so it runs on a throwaway copy of the tree.--- afterwards rpmspec reads the copy's spec, and whatever Version it says now--- is what `anda update` would have set+-- | anda rewrites specs in place, so it runs on a throwaway copy of the tree.+-- whatever Version the copy's spec says afterwards is what `anda update`+-- would have set module Oglyad.Source.Anda ( withScratchCopy, projectNames,+ reportSupported, updateProject,- scriptErrors,+ parseReport,+ ProjectRun (..),+ RunStatus (..),+ RunError (..),+ describeError, stripAnsi, ) where import Control.Exception (SomeException, finally, try)+import Data.Aeson (FromJSON (..), Value, eitherDecodeStrict, withObject, withText, (.!=), (.:), (.:?))+import Data.Aeson.Types (parseMaybe)+import Data.ByteString qualified as BS import Data.Map.Strict (Map) import Data.Map.Strict qualified as Map-import Data.Maybe (listToMaybe, mapMaybe)+import Data.Maybe (catMaybes, listToMaybe, mapMaybe) import Data.Text (Text) import Data.Text qualified as T import Oglyad.Process (run, runCapture) import Oglyad.Source.Cache (cacheDir)+import Oglyad.Types (SubmoduleEvent) import System.Directory (createDirectory, createDirectoryIfMissing, removeFile, removePathForcibly) import System.Exit (ExitCode (..)) import System.FilePath (takeDirectory, (</>)) import System.IO (hClose, openBinaryTempFile) --- | run an action on a copy of a tree (minus .git) that's deleted afterwards withScratchCopy :: FilePath -> (FilePath -> IO a) -> IO (Either Text a) withScratchCopy root act = do base <- (</> "anda-work") <$> cacheDir@@ -50,8 +55,7 @@ quietly :: IO () -> IO () quietly a = () <$ (try a :: IO (Either SomeException ())) --- | package directory to anda project name, from `anda list`, which prints--- lines like "anda/apps/foo/pkg (apps/foo)"+-- | `anda list` prints lines like "anda/apps/foo/pkg (apps/foo)" projectNames :: FilePath -> IO (Either Text (Map FilePath Text)) projectNames copy = do r <- runCapture copy [("NO_COLOR", "1")] "anda" ["list"]@@ -68,39 +72,94 @@ Just (takeDirectory (T.unpack path), name) _ -> Nothing --- | `anda update` one project in the copy. anda reports a failing script--- only in its log and still exits 0, so the log is where errors come from.--- the script gets GITHUB_TOKEN, since anda refuses gh() without one-updateProject :: FilePath -> Maybe Text -> Text -> Text -> IO (Either Text ())+reportSupported :: IO Bool+reportSupported = do+ r <- runCapture "." [("NO_COLOR", "1")] "anda" ["update", "--help"]+ pure $ case r of+ Right (ExitSuccess, out, _) -> "--report" `T.isInfixOf` out+ _ -> False++-- | anda still exits 0 when a script fails, so the report is where failures+-- come from. the script gets GITHUB_TOKEN, since anda refuses gh() without one+updateProject :: FilePath -> Maybe Text -> Text -> Text -> IO (Either Text [SubmoduleEvent]) updateProject copy token branch name = do+ (report, h) <- openBinaryTempFile copy "anda-report.json"+ hClose h let env = ("NO_COLOR", "1") : [("GITHUB_TOKEN", T.unpack t) | Just t <- [token]]- r <- runCapture copy env "anda" ["-v", "update", "--labels", "branch=" <> T.unpack branch, T.unpack name]- pure $ case r of- Left e -> Left e- Right (ExitFailure _, _, err) -> Left (failure (stripAnsi err))- Right (ExitSuccess, _, err) -> case scriptErrors (stripAnsi err) of- [] -> Right ()- e : _ -> Left e+ r <- runCapture copy env "anda" ["update", "--report", report, "--labels", "branch=" <> T.unpack branch, T.unpack name]+ case r of+ Left e -> pure (Left e)+ Right (ExitFailure _, _, err) -> pure (Left (failure (stripAnsi err)))+ Right (ExitSuccess, _, _) -> outcome . parseReport <$> BS.readFile report where -- color-eyre reports look like "Error:\n 0: Project not found: x" failure err = maybe (lastLine err) T.strip (listToMaybe [T.drop 3 l' | l <- T.lines err, let l' = T.stripStart l, "0: " `T.isPrefixOf` l'])+ outcome = \case+ Left e -> Left ("can't read anda's report: " <> e)+ Right [run'] -> case runStatus run' of+ Failed -> Left (maybe "update.rhai failed" describeError (runError run'))+ Panicked -> Left ("anda panicked" <> maybe "" ((": " <>) . errMessage) (runError run'))+ _ -> Right (runEvents run')+ Right runs -> Left ("anda's report has " <> T.pack (show (length runs)) <> " projects instead of one") --- | what went wrong in each script exception in anda's -v log: andax logs--- "ERROR andax::run: Script Exception —— <project>" and then the reason on a--- line starting with "├─═"-scriptErrors :: Text -> [Text]-scriptErrors = go . T.lines+data RunStatus = Ran | Failed | Terminated | Panicked+ deriving stock (Eq, Show)++instance FromJSON RunStatus where+ parseJSON = withText "status" $ \case+ "ok" -> pure Ran+ "failed" -> pure Failed+ "terminated" -> pure Terminated+ "panicked" -> pure Panicked+ other -> fail ("unknown status " <> T.unpack other)++data RunError = RunError+ { errMessage :: Text,+ errFunction :: Maybe Text,+ errLine :: Maybe Int+ }+ deriving stock (Eq, Show)++instance FromJSON RunError where+ parseJSON = withObject "error" $ \o ->+ RunError <$> o .: "message" <*> o .:? "function" <*> o .:? "line"++data ProjectRun = ProjectRun+ { runProject :: Text,+ runStatus :: RunStatus,+ runError :: Maybe RunError,+ runEvents :: [SubmoduleEvent]+ }+ deriving stock (Eq, Show)++instance FromJSON ProjectRun where+ parseJSON = withObject "project" $ \o ->+ ProjectRun+ <$> o .: "project"+ <*> o .: "status"+ <*> o .:? "error"+ <*> (mapMaybe (parseMaybe parseJSON) <$> (o .:? "events" .!= ([] :: [Value])))++newtype Report = Report [ProjectRun]++instance FromJSON Report where+ parseJSON = withObject "report" $ \o -> do+ version <- o .: "version"+ if version == (1 :: Int)+ then Report <$> o .: "projects"+ else fail ("unsupported report version " <> show version)++parseReport :: BS.ByteString -> Either Text [ProjectRun]+parseReport = either (Left . T.pack) (\(Report runs) -> Right runs) . eitherDecodeStrict++describeError :: RunError -> Text+describeError RunError {..} =+ T.unwords (catMaybes [(<> "():") <$> errFunction, Just (oneLine errMessage), lineNote <$> errLine]) where- go = \case- [] -> []- l : rest- | "Script Exception" `T.isInfixOf` l ->- case [T.strip (T.drop 1 (snd (T.breakOn "═" r))) | r <- take 5 rest, "├─═" `T.isInfixOf` r] of- msg : _ -> msg : go rest- [] -> "script exception" : go rest- | otherwise -> go rest+ lineNote n = "(line " <> T.pack (show n) <> ")"+ oneLine = T.intercalate ", " . filter (not . T.null) . map T.strip . T.lines --- | drop ANSI color codes, which color-eyre writes even with NO_COLOR set+-- | color-eyre writes ANSI color codes even with NO_COLOR set stripAnsi :: Text -> Text stripAnsi t = case T.breakOn "\ESC[" t of (pre, "") -> pre
src/Oglyad/Source/Cache.hs view
@@ -1,10 +1,9 @@--- | small json caches under $XDG_CACHE_HOME/oglyad, so reruns skip work that--- can't have changed: spec evaluation, repodata parsing and upstream lookups module Oglyad.Source.Cache ( cacheDir, loadJson, saveJson, CacheEntry (..),+ AndaEntry (..), UpstreamCache, lookupFresh, )@@ -19,7 +18,7 @@ import Data.Text qualified as T import Data.Time (NominalDiffTime, UTCTime, diffUTCTime) import GHC.Generics (Generic)-import Oglyad.Types (UpstreamSource)+import Oglyad.Types (SubmoduleEvent, UpstreamSource) import System.Directory (XdgDirectory (XdgCache), createDirectoryIfMissing, getXdgDirectory, renameFile) import System.FilePath ((</>)) import System.IO (hClose, openBinaryTempFile)@@ -27,31 +26,25 @@ cacheDir :: IO FilePath cacheDir = getXdgDirectory XdgCache "oglyad" --- | a missing or corrupt cache file is just a cache miss loadJson :: forall a. (FromJSON a) => FilePath -> IO (Maybe a) loadJson name = do path <- (</> name) <$> cacheDir r <- try (eitherDecodeFileStrict path) :: IO (Either SomeException (Either String a)) pure (either (const Nothing) (either (const Nothing) Just) r) --- | write to a temp file then rename, so a ctrl-c mid-write can't corrupt it.--- caching is best effort: if the write fails (full disk, read-only home) the--- run carries on and just won't be faster next time+-- | a unique temp file renamed into place, so a ctrl-c or a second run+-- writing at the same time can't leave a half-written cache behind saveJson :: (ToJSON a) => FilePath -> a -> IO ()-saveJson name x = do- r <- try $ do- dir <- cacheDir- createDirectoryIfMissing True dir- let path = dir </> name- -- a unique temp name, so two runs at once can't interleave their writes.- -- rename is atomic, so readers see the old file or the new one, never half- (tmp, h) <- openBinaryTempFile dir (name <> ".tmp")- LBS.hPut h (encode x) `finally` hClose h- renameFile tmp path- case r of- Left (_ :: SomeException) -> pure ()- Right () -> pure ()+saveJson name x = bestEffort $ do+ dir <- cacheDir+ createDirectoryIfMissing True dir+ (tmp, h) <- openBinaryTempFile dir (name <> ".tmp")+ LBS.hPut h (encode x) `finally` hClose h+ renameFile tmp (dir </> name) +bestEffort :: IO () -> IO ()+bestEffort action = either (\(_ :: SomeException) -> ()) id <$> try action+ data CacheEntry = CacheEntry { ceFetched :: UTCTime, ceResult :: Either Text Text@@ -59,8 +52,15 @@ deriving stock (Show, Generic) deriving anyclass (ToJSON, FromJSON) --- | upstream answers, keyed by the rendered 'UpstreamSource' so json keys--- stay plain strings+data AndaEntry = AndaEntry+ { aeFetched :: UTCTime,+ aeResult :: Either Text Text,+ aeEvents :: [SubmoduleEvent]+ }+ deriving stock (Show, Generic)+ deriving anyclass (ToJSON, FromJSON)++-- | keyed by the rendered 'UpstreamSource' so json keys stay plain strings type UpstreamCache = Map Text CacheEntry lookupFresh :: NominalDiffTime -> UTCTime -> UpstreamSource -> UpstreamCache -> Maybe CacheEntry
src/Oglyad/Source/Monorepo.hs view
@@ -1,9 +1,9 @@--- | reads a local checkout of terrapkg/packages, or another anda monorepo module Oglyad.Source.Monorepo ( PkgDir (..), SpecInfo (..), findPackageDirs, hclString,+ hclList, versionMacros, querySpec, lastCommits,@@ -23,6 +23,7 @@ import Data.ByteString.Base16 qualified as B16 import Data.Char (isAlphaNum, isSpace) import Data.List (sort)+import Data.List qualified as List import Data.Map.Strict (Map) import Data.Map.Strict qualified as Map import Data.Maybe (isJust, listToMaybe)@@ -41,25 +42,17 @@ import System.FilePath (takeDirectory, takeExtension, (</>)) import System.IO (hClose, openBinaryTempFile) --- | a directory with an anda.hcl in it data PkgDir = PkgDir- { -- | relative to the monorepo root+ { -- | relative to the monorepo root, like pdSpec pdPath :: FilePath,- -- | relative to the monorepo root pdSpec :: Maybe FilePath,- -- | `subrepo = "extras"` from the labels block pdSubrepo :: Maybe Text,- -- | file contents pdUpdateRhai :: Maybe Text,- -- | macros the spec's Version: line uses, like ["tag"] for- -- `Version: %(echo %tag | sed ...)`. Nothing when there's no spec or no- -- Version: line to read pdVersionMacros :: Maybe [Text],- -- | `oglyad_ignore = "check, other-check"` from the labels block pdIgnores :: [Text],- -- | has a `nightly = N` or `quaterdaily = N` label, so it builds from a- -- branch head on a schedule rather than from releases- pdNightly :: Bool+ pdNightly :: Bool,+ pdUpdBranch :: Bool,+ pdArches :: [Text] } deriving stock (Eq, Show) @@ -70,10 +63,7 @@ deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) --- | walk the package dir (anda/ in terrapkg/packages) and return every--- project directory. a directory that can't be read comes back as a Left--- with the reason, instead of ending the walk. symlinked directories aren't--- followed, so a link loop can't hang the walk+-- | symlinked directories aren't followed, so a link loop can't hang the walk findPackageDirs :: FilePath -> FilePath -> IO [Either (FilePath, Text) PkgDir] findPackageDirs root pkgRoot = go pkgRoot where@@ -84,8 +74,8 @@ Left (e :: IOException) -> pure [Left (rel, T.pack (show e))] Right entries -> do hasHcl <- doesFileExist (abs' </> "anda.hcl")- here <- if hasHcl then pure <$> readPkgDir rel entries else pure []- subdirs <- filterM (realDir . (abs' </>)) [e | e <- sort entries, take 1 e /= "."]+ here <- if hasHcl && not (null rel) then pure <$> readPkgDir rel entries else pure []+ subdirs <- filterM (nonSymlinkDir . (abs' </>)) [e | e <- sort entries, take 1 e /= "."] nested <- concat <$> forM subdirs (go . (rel </>)) pure (here <> nested) @@ -95,14 +85,14 @@ readPkgDir' rel entries = do let abs' = root </> rel- hcl <- readText (abs' </> "anda.hcl")+ hcl <- readUtf8Lenient (abs' </> "anda.hcl") let fromHcl = T.unpack <$> hclString "spec" hcl fallback = listToMaybe [e | e <- entries, takeExtension e == ".spec"] hasRhai <- doesFileExist (abs' </> "update.rhai")- rhai <- if hasRhai then Just <$> readText (abs' </> "update.rhai") else pure Nothing+ rhai <- if hasRhai then Just <$> readUtf8Lenient (abs' </> "update.rhai") else pure Nothing let spec = maybe fallback Just fromHcl specText <- case spec of- Just f -> either (\(_ :: IOException) -> Nothing) Just <$> try (readText (abs' </> f))+ Just f -> either (\(_ :: IOException) -> Nothing) Just <$> try (readUtf8Lenient (abs' </> f)) Nothing -> pure Nothing pure PkgDir@@ -112,13 +102,13 @@ pdUpdateRhai = rhai, pdVersionMacros = specText >>= versionMacros, pdIgnores = maybe [] (filter (not . T.null) . map T.strip . T.splitOn ",") (hclString "oglyad_ignore" hcl),- pdNightly = any (isJust . (`hclString` hcl)) ["nightly", "quaterdaily"]+ pdNightly = any (isJust . (`hclString` hcl)) ["nightly", "quaterdaily"],+ pdUpdBranch = isJust (hclString "updbranch" hcl),+ pdArches = hclList "arches" hcl } --- | the macro names a spec's Version: line refers to, read from the raw--- text: %name, %{name}, %{?name} and %{!?name:...} all count. rpmspec--- would expand them away, and what we want is which ones an update.rhai--- would have to set to change the version+-- | read from the raw text, since rpmspec would expand away the macros an+-- update.rhai has to set to change the version versionMacros :: Text -> Maybe [Text] versionMacros spec = case [T.strip v | l <- T.lines spec, Just v <- [stripTag l]] of v : _ -> Just (names v)@@ -134,8 +124,8 @@ (name, more) = T.span (\c -> isAlphaNum c || c == '_') after in [name | not (T.null name)] <> names more --- | the value of `key = "value"` (or a bare `key = 1`) anywhere in an--- anda.hcl. good enough for the flat labels and rpm blocks anda uses+-- | not a real HCL parser, but good enough for the flat labels and rpm+-- blocks anda uses hclString :: Text -> Text -> Maybe Text hclString key hcl = listToMaybe@@ -150,72 +140,75 @@ not (T.null v) ] --- | a directory that isn't a symlink-realDir :: FilePath -> IO Bool-realDir path = do+hclList :: Text -> Text -> [Text]+hclList key hcl =+ concat . take 1 $+ [ filter (not . T.null) (map (T.dropAround (\c -> c == '"' || isSpace c)) (T.splitOn "," inside))+ | l <- map T.strip (T.lines hcl),+ Just rest <- [T.stripPrefix key l],+ Just afterEq <- [T.stripPrefix "=" (T.stripStart rest)],+ Just opened <- [T.stripPrefix "[" (T.stripStart afterEq)],+ let inside = T.takeWhile (/= ']') opened+ ]++nonSymlinkDir :: FilePath -> IO Bool+nonSymlinkDir path = do isDir <- doesDirectoryExist path isLink <- either (\(_ :: IOException) -> False) id <$> try (pathIsSymbolicLink path) pure (isDir && not isLink) --- | read a text file as utf-8 no matter the locale, replacing invalid bytes--- instead of failing-readText :: FilePath -> IO Text-readText path = TE.decodeUtf8Lenient <$> BS.readFile path+readUtf8Lenient :: FilePath -> IO Text+readUtf8Lenient path = TE.decodeUtf8Lenient <$> BS.readFile path --- | let rpm evaluate the spec for us. hand-parsing specs means reimplementing--- the macro engine, which is a terrible idea-querySpec :: Int -> Text -> FilePath -> IO (Either Text SpecInfo)-querySpec release dist spec = do- let qf = "%{NAME}\t%|EPOCH?{%{EPOCH}}:{0}|:%{VERSION}-%{RELEASE}\n"+-- | rpm evaluates the spec, since hand-parsing specs would mean+-- reimplementing its macro engine+querySpec :: [(Text, Maybe Text)] -> FilePath -> IO (Either Text SpecInfo)+querySpec macros spec = do+ let nameAndEVR = "%{NAME}\t%|EPOCH?{%{EPOCH}}:{0}|:%{VERSION}-%{RELEASE}\n" out <- run "rpmspec" $- [ "-q",- "--srpm",- "--define",- "dist " <> T.unpack dist,- -- %fedora comes from the host otherwise, so -r 43 on an F44 box would- -- take the wrong side of every %if 0%{?fedora} >= 44- "--define",- "fedora " <> show release,- -- some specs load macro files from their own dir via %{_sourcedir}- "--define",- "_sourcedir " <> takeDirectory spec,- "--qf",- qf,- spec- ]+ ["-q", "--srpm"]+ <> concatMap macroArgs macros+ <> [ -- some specs load macro files from their own dir via %{_sourcedir}+ "--define",+ "_sourcedir " <> takeDirectory spec,+ "--qf",+ nameAndEVR,+ spec+ ] pure $ do o <- out case T.splitOn "\t" <$> take 1 (T.lines o) of [[name, evr]] | Just e <- parseEVR evr -> Right (SpecInfo name e) _ -> Left ("unexpected rpmspec output: " <> o) --- | time of the last commit touching each package dir, from a single--- `git log` over the whole history.------ asking git once per directory (`git log -1 -- dir`) makes it walk and diff--- history from the top every time, which for 1300 dirs and 20k commits burned--- ~25s of cpu. one pass listing every changed file per commit takes ~0.5s,--- and since git lists newest commits first, the first time we see a path--- under a dir is that dir's last commit+macroArgs :: (Text, Maybe Text) -> [String]+macroArgs = \case+ (name, Just body) -> ["--define", T.unpack name <> " " <> T.unpack body]+ (name, Nothing) -> ["--undefine", T.unpack name]++-- | one `git log` over the whole history instead of `git log -1 -- dir` per+-- directory, which walks history from the top every time (~25s of cpu for+-- 1300 dirs vs ~0.5s). git lists newest commits first, so the first time a+-- path under a dir shows up is that dir's last commit lastCommits :: FilePath -> FilePath -> Text -> [FilePath] -> IO (Map FilePath UTCTime) lastCommits root pkgRoot ref dirs = do out <- run "git" $- ["-C", root, "-c", "core.quotePath=false", "log", "--format=@%ct", "--name-only", "--no-renames", T.unpack ref, "--", pkgRoot]+ ["-C", root, "-c", "core.quotePath=false", "log", "--format=@%ct", "--name-only", "--no-renames", T.unpack ref, "--", if null pkgRoot then "." else pkgRoot] -- not a git checkout (or no git at all) just means no commit times pure (either (const Map.empty) (go Nothing Map.empty . T.lines) out) where wanted = Set.fromList (map T.pack dirs)- total = Set.size wanted- go _ acc _ | Map.size acc == total = acc -- every dir seen, stop reading+ everyDirSeen acc = Map.size acc == Set.size wanted+ go _ acc _ | everyDirSeen acc = acc go _ acc [] = acc go cur acc (l : ls) = case T.stripPrefix "@" l of -- unix seconds. iso dates would need care since git writes UTC as "Z", -- which the ZonedTime parser rejects Just stamp -> go (posixSecondsToUTCTime . fromInteger . fst <$> either (const Nothing) Just (TR.decimal stamp)) acc ls Nothing -> case cur of- Just t | not (T.null l) -> go cur (foldl' (claim t) acc (ancestors l)) ls+ Just t | not (T.null l) -> go cur (List.foldl' (claim t) acc (ancestors l)) ls _ -> go cur acc ls -- a file counts for every package dir above it, like `git log -- dir` does claim t acc d@@ -225,14 +218,11 @@ ("", _) -> [] (parentSlash, _) -> let parent = T.dropEnd 1 parentSlash in parent : ancestors parent --- | the commit HEAD points at, if this is a git checkout headCommit :: FilePath -> IO (Maybe Text) headCommit root = either (const Nothing) (Just . T.strip) <$> run "git" ["-C", root, "rev-parse", "HEAD"] --- | the whole tree of a git ref, extracted under the cache dir and keyed by--- commit, so each commit gets extracted once. the checkout itself is never--- touched: no checkout, no worktree, no fetch. returns the directory and the--- commit+-- | extracts into the cache once per commit, so the checkout itself is never+-- touched (no checkout, worktree or fetch) checkoutRef :: FilePath -> Text -> IO (Either Text (FilePath, Text)) checkoutRef mono ref = do rev <- run "git" ["-C", mono, "rev-parse", "--verify", "--quiet", T.unpack ref <> "^{commit}"]@@ -277,9 +267,8 @@ try' :: IO () -> IO () try' act = () <$ (try act :: IO (Either SomeException ())) --- | delete extracted trees nothing has used in a week, other than these.--- a week rather than right away, so checking one release doesn't throw out--- the trees the next full run needs+-- | trees are kept for a while after their last use, so checking one release+-- doesn't throw out the trees the next full run needs pruneTrees :: [Text] -> IO () pruneTrees keep = do trees <- (</> "trees") <$> cacheDir@@ -292,27 +281,27 @@ used <- try (getModificationTime path) :: IO (Either IOException UTCTime) case used of Right t- | length n == 40,+ | isCommitSha n, T.pack n `notElem` keep,- diffUTCTime now t > 7 * nominalDay ->+ diffUTCTime now t > keepUnusedFor -> () <$ (try (removePathForcibly path) :: IO (Either IOException ())) _ -> pure ()+ where+ isCommitSha n = length n == 40+ keepUnusedFor = 7 * nominalDay --- | a hash of everything that can change what rpmspec says about a package:--- the files in its directory and any subdirectories like patches/ (but not--- nested packages, which have their own fingerprint), the dist tag, and the+-- | covers everything that can change what rpmspec says: the package's files+-- (not nested packages, which have their own), the release's macros, and the -- rpm macros in effect (see 'macroFingerprint')-specFingerprint :: Text -> Text -> FilePath -> FilePath -> IO Text-specFingerprint macros dist root rel = do+specFingerprint :: Text -> [(Text, Maybe Text)] -> FilePath -> FilePath -> IO Text+specFingerprint macros defines root rel = do files <- packageFiles (root </> rel)- let header = TE.encodeUtf8 (macros <> "\0" <> dist <> "\0" <> T.pack rel <> "\0")- -- feed each file to the hash as its own chunk instead of concatenating- -- them, which would copy every byte of the package dir once more+ let header = TE.encodeUtf8 (macros <> "\0" <> T.pack (show defines) <> "\0" <> T.pack rel <> "\0")+ -- hashing file by file avoids copying the whole package dir into one buffer ctx <- foldM ( \c (f, entry) -> do body <- case entry of- -- hash where a symlink points, never what's behind it Link target -> pure ("<link> " <> TE.encodeUtf8 (T.pack target)) File -> either (\(e :: IOException) -> "<unreadable> " <> TE.encodeUtf8 (T.pack (show e))) id@@ -325,10 +314,8 @@ data Entry = File | Link FilePath --- | every file under a package dir, relative to it and sorted, skipping--- subdirectories that are packages of their own. symlinks are recorded with--- their target and never followed, so a dangling link or a link loop can't--- crash or hang the walk+-- | symlinks are recorded with their target and never followed, so a+-- dangling link or a link loop can't crash or hang the walk packageFiles :: FilePath -> IO [(FilePath, Entry)] packageFiles dir = go "" where@@ -348,10 +335,9 @@ (True, False) -> go rel _ -> pure [(rel, File)] --- | fingerprint of the whole rpm macro state, so updating rpm,--- redhat-rpm-config or anda-srpm-macros, or editing ~/.rpmmacros, invalidates--- cached evaluations. `rpm --showrc` dumps every macro rpm loaded, from every--- path in its search list, which is more complete than listing files by hand+-- | updating rpm, redhat-rpm-config or anda-srpm-macros, or editing+-- ~/.rpmmacros, invalidates cached evaluations. `rpm --showrc` dumps every+-- macro rpm loaded, which is more complete than listing macro files by hand macroFingerprint :: IO Text macroFingerprint = do showrc <- either ("no rpm: " <>) id <$> run "rpm" ["--showrc"]
src/Oglyad/Source/Repodata.hs view
@@ -1,5 +1,3 @@--- | reads published dnf repodata: metalink → repomd.xml →--- primary.xml.{zst,gz,xz} → packages module Oglyad.Source.Repodata ( RpmEntry (..), fetchRepo,@@ -47,23 +45,20 @@ import Text.XML.Cursor (attribute, attributeIs, content, element, fromDocument, ($/), ($//), (&/)) import Text.XML.Stream.Parse (parseBytes) --- | one binary (or source) rpm from primary.xml--- | strict fields, filled with copied text (see 'finish' in 'parsePrimary'),--- so a parsed entry never keeps the decompressed file alive+-- | strict fields with copied text (see 'finish' in 'parsePrimary'), so an+-- entry never keeps the decompressed file alive data RpmEntry = RpmEntry { rpmName :: !Text, rpmArch :: !Text, rpmEVR :: !EVR,- -- | e.g. zed-0.190.0-1.fc44.src.rpm, absent on src rpms+ -- | absent on src rpms rpmSourceRpm :: !(Maybe Text), rpmBuildTime :: !UTCTime } deriving stock (Eq, Show) --- | builds per source package for a repo, given its metalink or base url,--- reusing the last parse when the repo hasn't changed. repomd.xml is tiny,--- and the primary.xml href in it embeds a checksum, so an unchanged href--- means unchanged contents+-- | the primary.xml href in repomd.xml embeds a checksum, so an unchanged+-- href means the cached parse is still good fetchRepo :: Text -> App (Either String (Map Text RepoBuild)) fetchRepo url = do located <- fetchRepomd url@@ -81,10 +76,9 @@ Left _ -> pure () pure r --- | a repo's repomd.xml, and the base url it came from. a url with--- "metalink" in it is read like dnf reads one: try its mirrors by--- preference, and take the first repomd.xml whose sha256 the metalink--- lists, so a mirror that hasn't synced yet is skipped+-- | metalinks are read like dnf reads them: mirrors by preference, taking the+-- first repomd.xml whose sha256 the metalink lists, so a mirror that hasn't+-- synced yet is skipped fetchRepomd :: Text -> App (Either String (Text, LBS.ByteString)) fetchRepomd url | "metalink" `T.isInfixOf` url =@@ -105,9 +99,8 @@ | otherwise -> tryMirrors hashes (T.unpack m <> " doesn't match the metalink's checksum, is the mirror behind?") rest sha256 = TE.decodeUtf8 . B16.encode . SHA256.hashlazy --- | the sha256 hashes a metalink accepts for repomd.xml (the current one and--- any alternates, which are recent ones mirrors may still serve), and its--- mirrors' repomd.xml urls, most preferred first+-- | the hashes include alternates, recent repomd.xml versions mirrors may+-- still serve parseMetalink :: LBS.ByteString -> Either String ([Text], [Text]) parseMetalink bs = do doc <- first show (parseLBS def bs)@@ -123,7 +116,6 @@ ] Right (map T.strip hashes, map snd (sortOn (Down . fst) urls)) --- | fetch and parse the primary.xml at a href from repomd.xml fetchPrimaryAt :: Text -> Text -> App (Either String [RpmEntry]) fetchPrimaryAt base href = do download <- httpGetE (base <> "/" <> href)@@ -136,27 +128,24 @@ RateLimited why -> T.unpack why HttpFailed why -> T.unpack why --- | decompress a primary.xml by its file extension and parse it, streaming--- from the decompressor's pipe so even Fedora's ~250MB file never sits in--- memory.------ decompression goes through the zstd, gzip and xz commands, which every--- Fedora install has. Fedora doesn't package a Haskell zstd binding, and--- oglyad should build from Fedora's own libraries+-- | streams from the decompressor's pipe, so even Fedora's ~250MB file never+-- sits in memory. it shells out to zstd, gzip and xz because Fedora doesn't+-- package a Haskell zstd binding, and oglyad should build from Fedora's own+-- libraries decompressAndParse :: Text -> LBS.ByteString -> IO (Either String [RpmEntry]) decompressAndParse href compressed- | ".zst" `T.isSuffixOf` href = viaCommand "zstd" ["--decompress", "--stdout", "--quiet"]- | ".gz" `T.isSuffixOf` href = viaCommand "gzip" ["--decompress", "--stdout", "--quiet"]- | ".xz" `T.isSuffixOf` href = viaCommand "xz" ["--decompress", "--stdout", "--quiet"]+ | ".zst" `T.isSuffixOf` href = viaCommand "zstd"+ | ".gz" `T.isSuffixOf` href = viaCommand "gzip"+ | ".xz" `T.isSuffixOf` href = viaCommand "xz" | ".xml" `T.isSuffixOf` href = pure (parsePrimary compressed) | otherwise = pure (Left ("don't know how to decompress " <> T.unpack href)) where- -- feed the download to the command's stdin and parse its stdout as it- -- arrives. a corrupt archive makes the command fail, which shows up as- -- a parse error or a non-zero exit, never a crash- viaCommand cmd args = do+ -- a corrupt archive shows up as a parse error or a non-zero exit, never a+ -- crash+ viaCommand cmd = do env <- childEnv <$> getEnvironment- let cfg = setEnv env (setStdin (byteStringInput compressed) (setStdout createPipe (setStderr nullStream (proc cmd args))))+ let decompressToStdout = ["--decompress", "--stdout", "--quiet"]+ cfg = setEnv env (setStdin (byteStringInput compressed) (setStdout createPipe (setStderr nullStream (proc cmd decompressToStdout)))) r <- try $ withProcessTerm cfg $ \p -> do parsed <- parsePrimaryHandle (getStdout p) code <- waitExitCode p@@ -171,12 +160,7 @@ rpmNs n = Name n (Just "http://linux.duke.edu/metadata/rpm") Nothing metalinkNs n = Name n (Just "http://www.metalinker.org/") Nothing --- | find the location of primary.xml in repomd.xml.--- this uses the xml-conduit Cursor api, which builds the whole document in--- memory. that's fine for a 2KB file:--- $// = all descendants matching--- &/ = children matching--- >=> = compose two axes (it's just Kleisli composition on lists)+-- | builds the whole document in memory, which is fine for a 2KB file parseRepomd :: LBS.ByteString -> Either String Text parseRepomd bs = do doc <- first show (parseLBS def bs)@@ -188,7 +172,6 @@ (h : _) -> Right h [] -> Left "no primary entry in repomd.xml" --- | what we've seen so far of the <package> we're currently inside data PkgAcc = PkgAcc { accName :: Text, accArch :: Text,@@ -200,36 +183,26 @@ emptyAcc :: PkgAcc emptyAcc = PkgAcc "" "" Nothing Nothing Nothing --- | which element's text we're currently collecting data Capture = CapName | CapArch | CapSrpm | CapNone deriving stock (Eq) data St = St { stAcc :: !PkgAcc, stCap :: !Capture,- -- | finished packages, newest first stOut :: [RpmEntry],- -- | open elements, nonzero at the end means a truncated file- stDepth :: !Int+ stOpenElements :: !Int } --- | parse decompressed primary.xml from memory. the real runs stream from a--- pipe with 'parsePrimaryHandle'. this one is for small inputs and tests.------ unlike repomd.xml this file is big (Terra's is ~20MB, Fedora's is ~200MB)--- so we never build a DOM. instead the XML is turned into a stream of events--- (element opened, text, element closed, ...) and folded over with a small--- state machine. memory use stays proportional to the results, not the file+-- | for small inputs and tests. real runs stream with 'parsePrimaryHandle'.+-- primary.xml is too big for a DOM (Fedora's is ~200MB), so XML events are+-- folded over instead, keeping memory proportional to the results parsePrimary :: LBS.ByteString -> Either String [RpmEntry] parsePrimary bs = first (\e -> show (e :: SomeException)) (runConduit (sourceLazy bs .| parseBytes def .| foldlC stepPrimary st0)) >>= finishPrimary --- | parse primary.xml straight from a handle.------ reading the handle chunk by chunk matters: handing the parser a lazy--- ByteString instead lets the thunk evaluating it hold on to the start of the--- stream, which keeps the whole decompressed file (250MB for Fedora) alive+-- | reads chunk by chunk because a lazy ByteString would let its thunk hold+-- on to the start of the stream, keeping the whole decompressed file alive -- until parsing ends parsePrimaryHandle :: Handle -> IO (Either String [RpmEntry]) parsePrimaryHandle h = do@@ -241,14 +214,13 @@ finishPrimary :: St -> Either String [RpmEntry] finishPrimary final- | stDepth final /= 0 = Left "primary.xml ends in the middle of an element, truncated download?"+ | stOpenElements final /= 0 = Left "primary.xml ends in the middle of an element, truncated download?" | otherwise = Right (reverse (stOut final)) --- | one xml event. tracks element depth so truncation can be detected stepPrimary :: St -> Event -> St stepPrimary cur ev = case ev of- EventBeginElement _ _ -> step cur {stDepth = stDepth cur + 1} ev- EventEndElement _ -> step cur {stDepth = stDepth cur - 1} ev+ EventBeginElement _ _ -> step cur {stOpenElements = stOpenElements cur + 1} ev+ EventEndElement _ -> step cur {stOpenElements = stOpenElements cur - 1} ev _ -> step cur ev where step st = \case@@ -277,13 +249,9 @@ | otherwise -> st {stCap = CapNone} _ -> st - -- the text we collected is sliced out of the parser's buffers, and a- -- slice keeps its whole buffer alive. copying is what stops 75k Fedora- -- entries from pinning the entire decompressed file in memory- --- -- the build time needs the same care. UTCTime's fields are lazy, so- -- without a deep force it stays a thunk pointing at the xml attribute it- -- came from, which pins that attribute's whole decoder chunk+ -- collected text is a slice of the parser's buffers and keeps the whole+ -- buffer alive, so it's copied. UTCTime's fields are lazy, so the build+ -- time is deep forced for the same reason finish PkgAcc {..} = do EVR e v r <- accEVR t <- accTime@@ -305,8 +273,6 @@ Right (n, "") -> Just n _ -> Nothing --- | source package name from a srpm filename: strip ".src.rpm", then the last--- two dash-separated fields (version and release) srpmName :: Text -> Maybe Text srpmName file = do base <- T.stripSuffix ".src.rpm" file@@ -315,8 +281,6 @@ then Nothing else Just (T.intercalate "-" (take (length parts - 2) parts)) --- | collapse binary rpms down to one build per source package, keeping the--- newest build time. this is what lets us line repodata up with the monorepo latestBySource :: [RpmEntry] -> Map Text RepoBuild latestBySource = foldr step Map.empty where
src/Oglyad/Source/Rhai.hs view
@@ -1,17 +1,5 @@--- | figures out where an update.rhai gets its version from, without running it.------ evaluating Rhai would mean reimplementing andax, so instead we:------ 1. lex the script into tokens with megaparsec (strings, identifiers,--- punctuation, comments dropped)--- 2. find what's passed to `rpm.version(...)` or `rpm.global("ver", ...)`--- 3. if that's a variable, follow its `let` binding--- 4. if it's (or starts with) a known andax helper like gh("owner/repo"),--- that's the source------ versions that follow another distro or package (bodhi, madoguchi, nvidia)--- become 'Derived'. anything fancier, like custom get_json scraping, ends up as--- 'Unknown', and the unparsed-rhai check lists those+-- | figures out where an update.rhai gets its version from by lexing it,+-- since running it would mean reimplementing andax module Oglyad.Source.Rhai ( parseUpdateRhai, parseUpdateRhaiWith,@@ -19,11 +7,12 @@ tokenize, sourceCall, balanced,+ tracksSubmodules, ) where import Data.Char (isAlphaNum, isDigit, isSpace)-import Data.List (nub)+import Data.List (nub, tails) import Data.Maybe (listToMaybe, mapMaybe) import Data.Text (Text) import Data.Text qualified as T@@ -38,7 +27,7 @@ | TStr Text | TNum Text | TSym Char- | -- | a backtick string with ${...} in it, which Rhai interpolates+ | -- | a backtick string Rhai interpolates ${...} in TTmpl Text deriving stock (Eq, Show) @@ -47,10 +36,9 @@ parseUpdateRhai :: Text -> UpstreamSource parseUpdateRhai = parseUpdateRhaiWith Nothing --- | like 'parseUpdateRhai', also knowing which macros the spec's Version:--- line uses (see 'Oglyad.Source.Monorepo.versionMacros'). a script that--- sets none of them, and not ver or version either, doesn't track an--- upstream version at all+-- | a script that sets none of the macros the spec's Version: line uses (see+-- 'Oglyad.Source.Monorepo.versionMacros'), and not ver or version either,+-- doesn't track an upstream version at all parseUpdateRhaiWith :: Maybe [Text] -> Text -> UpstreamSource parseUpdateRhaiWith macros src = case tokenize src of Nothing -> Unknown (T.strip src)@@ -58,9 +46,9 @@ Just s -> s Nothing | hasCommitCall toks -> CommitTracked- -- a commit from somewhere else, like get(...).json().sha. only when- -- the script sets no version, since plenty record the commit of the- -- release they just scraped+ -- a commit from somewhere else, like get(...).json().sha. plenty of+ -- scripts also record the commit of the release they just scraped, so+ -- this only counts when no version is set | null (versionExprs toks), not (null (globalExprs ["commit"] toks)) -> CommitTracked | Just ms <- macros, null (versionExprs toks),@@ -71,8 +59,6 @@ | [s] <- nub (allSources toks) -> s | otherwise -> Unknown (T.strip src) --- * lexing- tokenize :: Text -> Maybe [Token] tokenize = either (const Nothing) Just . parse (sc *> many tokenP <* eof) "update.rhai" @@ -85,7 +71,7 @@ choice [ TStr <$> quoted '"', raw,- TStr <$> quoted '\'', -- char literals+ charLiteral, TIdent <$> takeWhile1P (Just "identifier") (\c -> isAlphaNum c || c == '_'), TNum <$> takeWhile1P (Just "number") isDigit, TSym <$> satisfy (not . isSpace)@@ -94,8 +80,8 @@ -- backtick strings are raw in Rhai, so regexes like `[\d.]+` stay intact. -- they interpolate ${...} though, so those get their own token raw = (\t -> if "${" `T.isInfixOf` t then TTmpl t else TStr t) <$> (char '`' *> takeWhileP Nothing (/= '`') <* char '`')- -- in quoted strings a backslash escapes whatever follows it. we only care- -- about the text being roughly right, so unknown escapes are kept as-is+ charLiteral = TStr <$> quoted '\''+ -- unknown escapes are kept as-is, the text only needs to be roughly right quoted q = T.pack <$> (char q *> manyTill (escaped <|> anySingle) (char q)) escaped = char '\\' *> fmap unescape anySingle unescape = \case@@ -103,8 +89,8 @@ 't' -> '\t' c -> c --- | do the brackets pair up? Rhai won't run a script where they don't, so--- this catches a broken update.rhai without a real parser+-- | Rhai won't run a script with unpaired brackets, so this catches a broken+-- update.rhai without a real parser balanced :: [Token] -> Bool balanced = go [] where@@ -118,36 +104,37 @@ _ : ts -> go stack ts pairs = [('(', ')'), ('[', ']'), ('{', '}')] --- * analysis+tracksSubmodules :: Text -> Bool+tracksSubmodules src = case tokenize src of+ Just toks -> any submodulesCall (tails toks)+ Nothing -> False+ where+ submodulesCall = \case+ TIdent "rpm" : TSym '.' : TIdent "submodules" : TSym '(' : _ -> True+ _ -> False --- | follow rpm.version(...) / rpm.global("ver", ...) back to a source helper resolveVersion :: [Token] -> Maybe UpstreamSource resolveVersion toks = listToMaybe (mapMaybe (resolveExpr toks 5) (versionExprs toks)) --- | every expression handed to something that sets the package version versionExprs :: [Token] -> [[Token]] versionExprs toks = go toks <> globalExprs ["ver", "version"] toks where go = \case [] -> []- -- rpm.version(EXPR)- TIdent "rpm" : TSym '.' : TIdent "version" : TSym '(' : rest -> takeBalanced rest : go rest+ TIdent "rpm" : TSym '.' : TIdent "version" : TSym '(' : rest -> takeArgument rest : go rest _ : rest -> go rest --- | every expression a script hands to rpm.global(KEY, ...) or--- rpm.define(KEY, ...) for one of these keys globalExprs :: [Text] -> [Token] -> [[Token]] globalExprs keys = \case [] -> [] TIdent "rpm" : TSym '.' : TIdent f : TSym '(' : TStr key : TSym ',' : rest | f `elem` ["global", "define"], key `elem` keys ->- takeBalanced rest : globalExprs keys rest+ takeArgument rest : globalExprs keys rest _ : rest -> globalExprs keys rest --- | tokens up to the ')' that closes the call we're inside-takeBalanced :: [Token] -> [Token]-takeBalanced = go (0 :: Int)+takeArgument :: [Token] -> [Token]+takeArgument = go (0 :: Int) where go _ [] = [] go depth (t : ts) = case t of@@ -157,15 +144,13 @@ | c == ',' && depth == 0 -> [] _ -> t : go depth ts --- | what does this expression get its value from? fuel stops let-cycles+-- | fuel stops let-cycles resolveExpr :: [Token] -> Int -> [Token] -> Maybe UpstreamSource resolveExpr toks fuel expr | fuel <= 0 = Nothing | otherwise = case expr of- -- a helper call, possibly with a method chain after it _ | Just (s, _) <- sourceCall expr -> Just s- -- a helper from one of the shared andax/*.rhai modules, e.g.- -- bump::madoguchi(...) or nvidia::nvidia_driver_version()+ -- helpers from the shared andax/*.rhai modules, like bump::madoguchi(...) TIdent m : TSym ':' : TSym ':' : TIdent f : TSym '(' : _ -> Just (Derived (m <> "::" <> f)) TIdent f : TSym '(' : _ | f `elem` derivedHelpers -> Just (Derived f) -- kmod packages copy Version from their kmod-common sibling with@@ -174,33 +159,28 @@ | "cat " `T.isPrefixOf` T.stripStart cmd, ".spec" `T.isInfixOf` cmd -> Just (Derived "another spec in the repo")- -- some other call like find(`regex`, tag, 1): the version comes from- -- whichever argument we can trace back to a source+ -- some other call like find(`regex`, tag, 1) TIdent _ : TSym '(' : rest -> listToMaybe (mapMaybe (resolveExpr toks (fuel - 1)) (splitArgs rest))- -- a variable, possibly with methods: `v` or `v.crop(1)` TIdent v : _ -> do- bound <- letBinding v toks+ bound <- firstLetBinding v toks resolveExpr toks (fuel - 1) bound _ -> Nothing --- | split the arguments of a call we're inside, up to its closing ')' splitArgs :: [Token] -> [[Token]]-splitArgs toks = case takeBalanced toks of+splitArgs toks = case takeArgument toks of [] -> [] arg -> arg : case drop (length arg) toks of TSym ',' : rest -> splitArgs rest _ -> [] --- | the right-hand side of `let v = ...;` (first binding wins)-letBinding :: Text -> [Token] -> Maybe [Token]-letBinding v = \case+firstLetBinding :: Text -> [Token] -> Maybe [Token]+firstLetBinding v = \case [] -> Nothing TIdent "let" : TIdent v' : TSym '=' : rest | v == v' -> Just (takeWhile (/= TSym ';') rest)- _ : rest -> letBinding v rest+ _ : rest -> firstLetBinding v rest --- | recognize an andax source helper at the start of a token list sourceCall :: [Token] -> Maybe (UpstreamSource, [Token]) sourceCall = \case TIdent f : TSym '(' : TStr a : TSym ',' : TStr b : TSym ')' : rest@@ -239,7 +219,6 @@ "ansible_galaxy" -> Just (AnsibleGalaxy a b) _ -> Nothing --- | helpers whose version follows something other than an upstream release derivedHelpers :: [Text] derivedHelpers = [ "bodhi",@@ -256,7 +235,6 @@ [] -> [] ts@(_ : rest) -> maybe id ((:) . fst) (sourceCall ts) (allSources rest) --- | does the script call any *_commit helper (gh_commit, gitlab_commit, ...)? hasCommitCall :: [Token] -> Bool hasCommitCall = \case TIdent f : TSym '(' : _ | "_commit" `T.isSuffixOf` f -> True
src/Oglyad/Source/RhaiEval.hs view
@@ -1,11 +1,7 @@--- | runs the version-setting part of an update.rhai against an upstream value--- we already fetched, so guards like `if v.starts_with("cli-v")` and edits like--- `v.crop(5)` apply before drift compares it with the spec.------ it understands a small slice of Rhai: let, assignment, if/else, try/catch,--- terminate(), rpm.version / rpm.global("ver", ...), the string methods update--- scripts use, andax's regex helpers (find, find_all, sub), template strings,--- + and indexing. anything else that touches a value we need gives up with+-- | replays an update.rhai against an upstream value we already fetched, so+-- guards like `if v.starts_with("cli-v")` and edits like `v.crop(5)` apply+-- before drift compares it with the spec. it only models the slice of Rhai+-- update scripts actually use. anything else that touches a value we need is -- 'CantTell', and drift falls back to the raw upstream value module Oglyad.Source.RhaiEval ( Simulated (..),@@ -26,18 +22,14 @@ import Text.Read (readMaybe) data Simulated- = -- | the script would set Version to this- Sets Text- | -- | the script finishes (or calls terminate()) without setting Version,- -- so `anda update` wouldn't touch the spec for this upstream value+ = Sets Text+ | -- | `anda update` wouldn't touch the spec for this upstream value NoUpdate- | -- | the script does something we don't model- CantTell+ | CantTell deriving stock (Eq, Show) --- | what Version the script would set, given the value its source helper--- returns. the source is the one 'Oglyad.Source.Rhai.parseUpdateRhai' found.--- any other helper call evaluates to an unknown value+-- | only calls to the source 'Oglyad.Source.Rhai.parseUpdateRhai' found get+-- the upstream value. any other helper call is unknown simulateVersion :: UpstreamSource -> Text -> Text -> Simulated simulateVersion src upstream script = case tokenize script of Just toks@@ -52,19 +44,14 @@ Left GiveUp -> CantTell _ -> CantTell where- -- rpm.version wins over rpm.global("ver", ...), since some scripts set the- -- global to the raw tag and Version to a cleaned-up spelling of it outcome env = maybe NoUpdate Sets (envVersion env <|> envGlobal env) --- * syntax- data Value = VStr Text | VNum Int | VBool Bool | VArr [Value] | VUnit deriving stock (Eq, Show) data Expr = Lit Value- | -- | a backtick string with ${...} to fill in- Tmpl Text+ | Template Text | Var Text | Src UpstreamSource | Call Text [Expr]@@ -74,32 +61,33 @@ | Bin Text Expr Expr | -- | rpm.changed(). we're asking "what if upstream moved", so it's true Changed- | -- | an expression we couldn't parse. only fatal if something needs it+ | -- | only fatal if something needs its value Unparsed deriving stock (Show) +-- | rpm.version wins over rpm.global("ver", ...), since some scripts set the+-- global to the raw tag and Version to a cleaned-up spelling of it+data VersionTarget = RpmVersion | RpmGlobal+ deriving stock (Show)+ data Stmt = Let Text Expr- | -- | v.crop(1); and friends, which change v in place- Mutate Text Text [Expr]- | -- | True for rpm.version, False for rpm.global("ver", ...)- SetVersion Bool Expr+ | Mutate Text Text [Expr]+ | SetVersion VersionTarget Expr | If Expr [Stmt] [Stmt] | Try [Stmt] [Stmt] | Terminate- | -- | rpm.release(), print(...), and other calls that can't change Version+ | -- | calls that can't change Version, like rpm.release() and print(...) Ignored | -- | an expression on its own, like find(...) used as a guard inside try ExprStmt Expr [Token]- | -- | a statement we don't understand, kept so we can see what it touches+ | -- | kept so we can see what it touches Opaque [Token] | -- | `rpm.f = ...`, which rewrites the whole spec and can change Version -- in ways we can't follow RewritesSpec deriving stock (Show) --- * parsing- stmts :: [Token] -> Maybe [Stmt] stmts = \case [] -> Just []@@ -112,7 +100,7 @@ stmt = \case TIdent "if" : rest -> ifStmt rest TIdent "try" : TSym '{' : rest -> do- (body, rest1) <- block rest+ (body, rest1) <- blockAfterBrace rest case rest1 of TIdent "catch" : rest2 -> do let rest3 = case rest2 of@@ -120,12 +108,12 @@ r -> r case rest3 of TSym '{' : r -> do- (handler, rest4) <- block r+ (handler, rest4) <- blockAfterBrace r Just (Try body handler, rest4) _ -> Nothing _ -> Nothing toks ->- let (s, rest) = untilSemi toks+ let (s, rest) = splitAtTopLevelSemicolon toks in Just (simple s, rest) ifStmt :: [Token] -> Maybe (Stmt, [Token])@@ -134,20 +122,19 @@ rest1 <- case rest of TSym '{' : r -> Just r _ -> Nothing- (yes, rest2) <- block rest1+ (yes, rest2) <- blockAfterBrace rest1 case rest2 of TIdent "else" : TIdent "if" : rest3 -> do (s, rest4) <- ifStmt rest3 Just (If (expr c) yes [s], rest4) TIdent "else" : TSym '{' : rest3 -> do- (no, rest4) <- block rest3+ (no, rest4) <- blockAfterBrace rest3 Just (If (expr c) yes no, rest4) _ -> Just (If (expr c) yes [], rest2) --- | the statements inside braces, given the tokens just after the '{'-block :: [Token] -> Maybe ([Stmt], [Token])-block toks = do- (inner, rest) <- closing toks+blockAfterBrace :: [Token] -> Maybe ([Stmt], [Token])+blockAfterBrace toks = do+ (inner, rest) <- splitAfterOpenBracket toks body <- stmts inner Just (body, rest) @@ -156,36 +143,33 @@ [TIdent "terminate", TSym '(', TSym ')'] -> Terminate TIdent "let" : TIdent v : TSym '=' : e -> Let v (expr e) TIdent "rpm" : TSym '.' : TIdent "version" : TSym '(' : r- | Just [a] <- argsOnly r -> SetVersion True a+ | Just [a] <- finalCallArgs r -> SetVersion RpmVersion a TIdent "rpm" : TSym '.' : TIdent f : TSym '(' : TStr key : TSym ',' : r | f `elem` ["global", "define"], key `elem` ["ver", "version"],- Just [a] <- argsOnly r ->- SetVersion False a+ Just [a] <- finalCallArgs r ->+ SetVersion RpmGlobal a TIdent "rpm" : TSym '.' : TIdent "f" : TSym '=' : t : _ | t /= TSym '=' -> RewritesSpec TIdent "rpm" : _ -> Ignored TIdent "print" : _ -> Ignored TIdent v : TSym '.' : TIdent m : TSym '(' : r- | m `elem` mutators,- Just as <- argsOnly r ->+ | m `elem` inPlaceStringMethods,+ Just as <- finalCallArgs r -> Mutate v m as TIdent v : TSym '=' : e@(t : _) | t /= TSym '=' -> Let v (expr e) toks -> case expr toks of Unparsed -> Opaque toks e -> ExprStmt e toks where- -- the arguments of a call that ends the statement- argsOnly r = case args r of+ finalCallArgs r = case argsAfterParen r of Just (as, []) -> Just as _ -> Nothing --- | string methods that change the string they're called on-mutators :: [Text]-mutators = ["crop", "truncate", "pop", "replace", "trim", "make_lower", "make_upper"]+inPlaceStringMethods :: [Text]+inPlaceStringMethods = ["crop", "truncate", "pop", "replace", "trim", "make_lower", "make_upper"] --- | tokens up to a ';' outside any brackets, and what's after it-untilSemi :: [Token] -> ([Token], [Token])-untilSemi = go (0 :: Int)+splitAtTopLevelSemicolon :: [Token] -> ([Token], [Token])+splitAtTopLevelSemicolon = go (0 :: Int) where go _ [] = ([], []) go d (t : ts) = case t of@@ -196,10 +180,8 @@ _ -> cons t (go d ts) cons t (a, b) = (t : a, b) --- | given the tokens after an opening bracket, the tokens inside it and the--- ones after its closing bracket-closing :: [Token] -> Maybe ([Token], [Token])-closing = go (0 :: Int)+splitAfterOpenBracket :: [Token] -> Maybe ([Token], [Token])+splitAfterOpenBracket = go (0 :: Int) where go _ [] = Nothing go d (t : ts) = case t of@@ -209,10 +191,9 @@ _ -> cons t (go d ts) cons t = fmap (\(a, b) -> (t : a, b)) --- | call arguments, given the tokens after the '('-args :: [Token] -> Maybe ([Expr], [Token])-args toks = do- (inner, rest) <- closing toks+argsAfterParen :: [Token] -> Maybe ([Expr], [Token])+argsAfterParen toks = do+ (inner, rest) <- splitAfterOpenBracket toks Just (if null inner then [] else map expr (splitCommas inner), rest) where splitCommas ts = case go (0 :: Int) ts of@@ -227,7 +208,7 @@ _ -> cons t (go d ts) cons t (a, b) = (t : a, b) --- | a whole expression. leftover tokens mean we misread it+-- | leftover tokens mean we misread it expr :: [Token] -> Expr expr toks = case orE toks of Just (e, []) -> e@@ -265,7 +246,7 @@ postfix p r primary = \case TStr s : r -> Just (Lit (VStr s), r)- TTmpl s : r -> Just (Tmpl s, r)+ TTmpl s : r -> Just (Template s, r) TNum n : r -> (\i -> (Lit (VNum i), r)) <$> readMaybe (T.unpack n) TIdent "true" : r -> Just (Lit (VBool True), r) TIdent "false" : r -> Just (Lit (VBool False), r)@@ -280,21 +261,20 @@ TIdent n : r | T.all isDigit n -> (\i -> (Lit (VNum i), r)) <$> readMaybe (T.unpack n) TSym '-' : TIdent n : r | T.all isDigit n -> (\i -> (Lit (VNum (negate i)), r)) <$> readMaybe (T.unpack n) TIdent f : TSym '(' : r -> do- (as, r') <- args r+ (as, r') <- argsAfterParen r Just (Call f as, r') TIdent v : r@(t : _) | t /= TSym ':' -> Just (Var v, r) [TIdent v] -> Just (Var v, []) _ -> Nothing --- | method calls, properties and indexing: v.crop(1), v.len, caps[1] postfix :: Expr -> P postfix e = \case TSym '.' : TIdent m : TSym '(' : r -> do- (as, r') <- args r+ (as, r') <- argsAfterParen r postfix (Method e m as) r' TSym '.' : TIdent m : r -> postfix (Method e m []) r TSym '[' : r -> do- (inner, r') <- closing r+ (inner, r') <- splitAfterOpenBracket r postfix (Index e (expr inner)) r' r -> Just (e, r) @@ -312,8 +292,6 @@ stripPrefixList (x : xs) (y : ys) | x == y = stripPrefixList xs ys stripPrefixList _ _ = Nothing --- * evaluation- data Env = Env { -- | Nothing is a value we can't know, like another helper's result envVars :: Map Text (Maybe Value),@@ -326,16 +304,14 @@ data Stop = Terminated Env- | -- | a Rhai error, like find() not matching. try/catch can stop it+ | -- | a Rhai error, like find() not matching Thrown Env | GiveUp --- | why an expression has no value data NoValue- = -- | it depends on something we can't see or don't model+ = -- | depends on something we can't see or don't model Unknown- | -- | evaluating it raises a Rhai error- Throw+ | Throw type Eval = Either NoValue Value @@ -357,17 +333,17 @@ Left Unknown -> Right (bind v Nothing) -- mutating something we can't see keeps it unknown _ -> Right env- SetVersion isMain e -> case eval ctx env e of- Right (VStr t)- | isMain -> Right env {envVersion = Just t}- | otherwise -> Right env {envGlobal = Just t}+ SetVersion target e -> case eval ctx env e of+ Right (VStr t) -> Right $ case target of+ RpmVersion -> env {envVersion = Just t}+ RpmGlobal -> env {envGlobal = Just t} Left Throw -> Left (Thrown env) _ -> Left GiveUp If c yes no -> case eval ctx env c of Right (VBool b) -> exec ctx env (if b then yes else no) Left Throw -> Left (Thrown env) -- a branch we can't decide only matters if it could change Version- _ | all inert (yes <> no) -> Right env+ _ | all cannotChangeVersion (yes <> no) -> Right env _ -> Left GiveUp Try body handler -> case exec ctx env body of Left (Thrown env') -> exec ctx env' handler@@ -389,18 +365,17 @@ touches toks = any known [v | TIdent v <- toks] known v = maybe False isJust (Map.lookup v (envVars env)) --- | can this statement affect what Version ends up as?-inert :: Stmt -> Bool-inert = \case+cannotChangeVersion :: Stmt -> Bool+cannotChangeVersion = \case Ignored -> True- If _ yes no -> all inert (yes <> no)- Try body handler -> all inert (body <> handler)+ If _ yes no -> all cannotChangeVersion (yes <> no)+ Try body handler -> all cannotChangeVersion (body <> handler) _ -> False eval :: Ctx -> Env -> Expr -> Eval eval ctx@(src, upstream) env = \case Lit v -> Right v- Tmpl t -> VStr <$> template t+ Template t -> VStr <$> template t Var v -> maybe (Left Unknown) Right (Map.lookup v (envVars env) >>= id) Src s | s == src -> Right (VStr upstream)@@ -442,32 +417,31 @@ ((pre <> shown) <>) <$> template (T.drop 1 after) _ -> Left Unknown --- | andax functions. regexes the engine can't compile are unknown rather--- than errors, since that's usually our engine's limit and not the script's+-- | andax functions. regexes the engine can't compile or run are unknown+-- rather than errors, since that's usually our engine's limit and not the+-- script's call :: Text -> [Value] -> Eval call f as = case (f, as) of ("find", [VStr r, VStr t, VNum g]) -> do re <- regex r- caps <- slow (Re.captures re t) >>= maybe (Left Throw) Right+ caps <- engineLimitIsUnknown (Re.captures re t) >>= maybe (Left Throw) Right if g < 0 || g >= length caps then Left Throw else maybe (Left Throw) (Right . VStr) (caps !! g) ("find_all", [VStr r, VStr t]) -> do re <- regex r- all' <- slow (Re.capturesAll re t)+ all' <- engineLimitIsUnknown (Re.capturesAll re t) Right (VArr [VArr (map (maybe VUnit VStr) caps) | caps <- all']) ("sub", [VStr r, VStr rep, VStr t]) -> do re <- regex r- VStr <$> slow (Re.replaceAll re rep t)+ VStr <$> engineLimitIsUnknown (Re.replaceAll re rep t) _ -> Left Unknown where- regex = either (const (Left Unknown)) Right . Re.compile- -- Rust would have answered, we just ran out of patience- slow = either (const (Left Unknown)) Right+ regex = engineLimitIsUnknown . Re.compile+ engineLimitIsUnknown = either (const (Left Unknown)) Right --- | built-in methods that don't change the value they're called on. the--- in-place ones return the changed string here, which is how scripts like--- `gh(...).crop(1)` read+-- | in-place methods return the changed string here, which is how scripts+-- like `gh(...).crop(1)` read method :: Value -> Text -> [Value] -> Eval method val m as = case (val, m, as) of (VStr s, "pop", []) -> Right (VStr (T.takeEnd 1 s))@@ -483,8 +457,8 @@ (VArr xs, "is_empty", []) -> Right (VBool (null xs)) _ -> mutate val m as --- | what an in-place string method leaves behind, following Rhai's string--- api (crop keeps from start, truncate keeps the first n, pop drops the end)+-- | follows Rhai's string api: crop keeps from start, truncate keeps the+-- first n, pop drops the end mutate :: Value -> Text -> [Value] -> Eval mutate (VStr s) m as = VStr <$> case (m, as) of@@ -500,8 +474,7 @@ _ -> Left Unknown mutate _ _ _ = Left Unknown --- | arrays and strings, with negative indices counting from the end like--- Rhai. out of range is a Rhai error+-- | negative indices count from the end, like Rhai index :: Value -> Value -> Eval index val (VNum i) = case val of VArr xs -> pick xs
src/Oglyad/Source/Upstream.hs view
@@ -13,6 +13,7 @@ import Data.Aeson.KeyMap qualified as KM import Data.Bifunctor (first) import Data.Char (isAlphaNum, isDigit)+import Data.Function (on) import Data.List (maximumBy) import Data.Text (Text) import Data.Text qualified as T@@ -22,30 +23,28 @@ import Oglyad.Types (CrateField (..), UpstreamSource (..)) import Oglyad.Version (normalizeVersion, rpmvercmp) --- | a step into a json document: an object key or an array index-data Step = K Text | I Int+data JsonStep = Field Text | Index Int --- | latest upstream version, or a reason we couldn't get it latestVersion :: UpstreamSource -> App (Either HttpError Text) latestVersion = \case- GitHub repo -> json ("https://api.github.com/repos/" <> repo <> "/releases/latest") [K "tag_name"]- GitHubTag repo -> json ("https://api.github.com/repos/" <> repo <> "/tags?per_page=1") [I 0, K "name"]- GitLab host i -> json ("https://" <> host <> "/api/v4/projects/" <> i <> "/releases/") [I 0, K "tag_name"]- GitLabTag host i -> json ("https://" <> host <> "/api/v4/projects/" <> i <> "/repository/tags") [I 0, K "name"]- Gitea host repo -> json ("https://" <> host <> "/api/v1/repos/" <> repo <> "/releases/latest") [K "tag_name"]- GiteaTag host repo -> json ("https://" <> host <> "/api/v1/repos/" <> repo <> "/tags") [I 0, K "name"]- PyPI name -> json ("https://pypi.org/pypi/" <> name <> "/json") [K "info", K "version"]- Crates field name -> json ("https://crates.io/api/v1/crates/" <> name) [K "crate", K (crateKey field)]- Npm name -> json ("https://registry.npmjs.org/" <> name <> "/latest") [K "version"]- Hackage name -> json ("https://hackage.haskell.org/package/" <> name <> "/preferred.json") [K "normal-version", I 0]- Gems name -> json ("https://rubygems.org/api/v1/versions/" <> name <> "/latest.json") [K "version"]- Hex name -> json ("https://hex.pm/api/packages/" <> name) [K "latest_stable_version"]- Cran name -> json ("https://crandb.r-pkg.org/" <> name) [K "Version"]+ GitHub repo -> jsonAt ("https://api.github.com/repos/" <> repo <> "/releases/latest") [Field "tag_name"]+ GitHubTag repo -> jsonAt ("https://api.github.com/repos/" <> repo <> "/tags?per_page=1") [Index 0, Field "name"]+ GitLab host i -> jsonAt ("https://" <> host <> "/api/v4/projects/" <> i <> "/releases/") [Index 0, Field "tag_name"]+ GitLabTag host i -> jsonAt ("https://" <> host <> "/api/v4/projects/" <> i <> "/repository/tags") [Index 0, Field "name"]+ Gitea host repo -> jsonAt ("https://" <> host <> "/api/v1/repos/" <> repo <> "/releases/latest") [Field "tag_name"]+ GiteaTag host repo -> jsonAt ("https://" <> host <> "/api/v1/repos/" <> repo <> "/tags") [Index 0, Field "name"]+ PyPI name -> jsonAt ("https://pypi.org/pypi/" <> name <> "/json") [Field "info", Field "version"]+ Crates field name -> jsonAt ("https://crates.io/api/v1/crates/" <> name) [Field "crate", Field (crateKey field)]+ Npm name -> jsonAt ("https://registry.npmjs.org/" <> name <> "/latest") [Field "version"]+ Hackage name -> jsonAt ("https://hackage.haskell.org/package/" <> name <> "/preferred.json") [Field "normal-version", Index 0]+ Gems name -> jsonAt ("https://rubygems.org/api/v1/versions/" <> name <> "/latest.json") [Field "version"]+ Hex name -> jsonAt ("https://hex.pm/api/packages/" <> name) [Field "latest_stable_version"]+ Cran name -> jsonAt ("https://crandb.r-pkg.org/" <> name) [Field "Version"] Opam name -> opam name AnsibleGalaxy ns coll ->- json+ jsonAt ("https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/index/" <> ns <> "/" <> coll <> "/versions/?limit=1&ordering=-version")- [K "data", I 0, K "version"]+ [Field "data", Index 0, Field "version"] GitTags url -> first HttpFailed <$> liftIO (gitTags url) Derived how -> pure (Left (HttpFailed ("follows " <> how <> ", not an upstream release"))) CommitTracked -> pure (Left (HttpFailed "tracks a commit, not a version"))@@ -57,9 +56,8 @@ Max -> "max_version" Newest -> "newest_version" --- | GET a url and walk a path into the json it returns-json :: Text -> [Step] -> App (Either HttpError Text)-json url path = do+jsonAt :: Text -> [JsonStep] -> App (Either HttpError Text)+jsonAt url path = do body <- httpGetE url pure $ do v <- body >>= either (\e -> Left (HttpFailed ("bad json from " <> url <> ": " <> T.pack e))) Right . eitherDecode@@ -67,10 +65,10 @@ Just (String s) | not (T.null s) -> Right s _ -> Left (HttpFailed ("no version at expected spot in " <> url)) -walk :: [Step] -> Value -> Maybe Value+walk :: [JsonStep] -> Value -> Maybe Value walk [] v = Just v-walk (K k : rest) (Object o) = KM.lookup (Key.fromText k) o >>= walk rest-walk (I i : rest) (Array a) = a V.!? i >>= walk rest+walk (Field k : rest) (Object o) = KM.lookup (Key.fromText k) o >>= walk rest+walk (Index i : rest) (Array a) = a V.!? i >>= walk rest walk _ _ = Nothing -- | opam has no "latest" endpoint. anda lists the package's directory in@@ -93,7 +91,7 @@ _ -> [] case versions of [] -> Left (HttpFailed ("no versions listed at " <> url))- _ -> Right (maximumBy (\a b -> rpmvercmp (normalizeVersion a) (normalizeVersion b)) versions)+ _ -> Right (newest versions) -- | does this parse with the semver crate after anda's opam cleanup (leading -- v's dropped, '~' read as '-')? MAJOR.MINOR.PATCH with no leading zeros,@@ -111,13 +109,18 @@ [a, b, c] -> all numeric [a, b, c] && idents preIdent pre && idents ident build _ -> False --- | newest tag on a git remote, for forges without a releases api.--- `git ls-remote` speaks the smart http protocol, so nothing gets cloned+-- | `git ls-remote` speaks the smart http protocol, so nothing gets cloned gitTags :: Text -> IO (Either Text Text) gitTags url = do out <- run "git" ["ls-remote", "--tags", "--refs", T.unpack url] pure $ case out of Left err -> Left ("git ls-remote failed for " <> url <> ": " <> err)- Right o -> case [t | l <- T.lines o, Just t <- [T.stripPrefix "refs/tags/" (T.takeWhileEnd (/= '\t') l)]] of+ Right refs -> case tagNames refs of [] -> Left ("no tags at " <> url)- tags -> Right (maximumBy (\a b -> rpmvercmp (normalizeVersion a) (normalizeVersion b)) tags)+ tags -> Right (newest tags)++tagNames :: Text -> [Text]+tagNames lsRemote = [tag | l <- T.lines lsRemote, Just tag <- [T.stripPrefix "refs/tags/" (T.takeWhileEnd (/= '\t') l)]]++newest :: [Text] -> Text+newest = maximumBy (rpmvercmp `on` normalizeVersion)
src/Oglyad/Types.hs view
@@ -4,6 +4,7 @@ CrateField (..), Package (..), RepoBuild (..),+ SubmoduleEvent (..), Severity (..), Finding (..), finding,@@ -14,7 +15,7 @@ ) where -import Data.Aeson (FromJSON, ToJSON (..), Value (String))+import Data.Aeson (FromJSON (..), ToJSON (..), Value (String), object, withObject, (.:), (.=)) import Data.Map.Strict (Map) import Data.Set (Set) import Data.Text (Text)@@ -23,7 +24,6 @@ import GHC.Generics (Generic) import Oglyad.Version (EVR (..)) --- | which version field of a crate anda reads data CrateField = -- | crates() MaxStable@@ -34,21 +34,16 @@ deriving stock (Eq, Ord, Show, Read, Generic) deriving anyclass (ToJSON, FromJSON) --- | where a package's upstream version comes from, recovered from update.rhai.--- constructors mirror the andax helpers in FyraLabs/anda (andax/fns/tsunagu.rs)+-- | mirrors the andax helpers in FyraLabs/anda (andax/fns/tsunagu.rs) data UpstreamSource- = -- | gh("owner/repo"), latest release- GitHub Text- | -- | gh_tag("owner/repo"), newest tag- GitHubTag Text- | -- | gitlab([domain,] id), latest release+ = GitHub Text+ | GitHubTag Text+ | -- | domain, project id GitLab Text Text- | -- | gitlab_tag([domain,] id)- GitLabTag Text Text- | -- | gitea(host, repo) and codeberg(repo)+ | GitLabTag Text Text+ | -- | host, repo. codeberg() lands here too Gitea Text Text- | -- | gitea_tag(host, repo) and codeberg_tag(repo)- GiteaTag Text Text+ | GiteaTag Text Text | PyPI Text | Crates CrateField Text | Npm Text@@ -59,21 +54,17 @@ | Opam Text | -- | namespace, collection AnsibleGalaxy Text Text- | -- | newest tag of any git remote (sourcehut, tangled)- GitTags Text+ | GitTags Text | -- | follows another distro or package (bodhi, madoguchi, nvidia), not upstream Derived Text- | -- | follows a branch head (gh_commit and friends), no version to compare+ | -- | follows a branch head (gh_commit and friends), so there's no version to compare CommitTracked- | -- | never sets anything the spec's Version: is built from. it only bumps- -- Release (say, to rebuild when something else changes) or writes files+ | -- | the script only bumps Release or writes files, never the Version: inputs NoVersion- | -- | raw update.rhai we couldn't make sense of- Unknown Text+ | Unknown Text deriving stock (Eq, Ord, Show, Read, Generic) deriving anyclass (ToJSON, FromJSON) --- | what Terra's published repodata says about a source package data RepoBuild = RepoBuild { rbEVR :: EVR, rbBuildTime :: UTCTime@@ -81,36 +72,50 @@ deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON, FromJSON) --- | one source package, merged from every data source we know about+data SubmoduleEvent+ = SubmoduleAdded {sePath :: Text, seUrl :: Text, seCommit :: Text, seGlobal :: Text}+ | SubmoduleDropped {seGlobal :: Text, seCommit :: Text}+ deriving stock (Eq, Show)++instance FromJSON SubmoduleEvent where+ parseJSON = withObject "event" $ \o ->+ o .: "kind" >>= \case+ "submodule-added" -> SubmoduleAdded <$> o .: "path" <*> o .: "url" <*> o .: "commit" <*> o .: "global"+ "submodule-dropped" -> SubmoduleDropped <$> o .: "global" <*> o .: "commit"+ other -> fail ("unknown event " <> T.unpack other)++instance ToJSON SubmoduleEvent where+ toJSON = \case+ SubmoduleAdded {..} ->+ object ["kind" .= ("submodule-added" :: Text), "path" .= sePath, "url" .= seUrl, "commit" .= seCommit, "global" .= seGlobal]+ SubmoduleDropped {..} ->+ object ["kind" .= ("submodule-dropped" :: Text), "global" .= seGlobal, "commit" .= seCommit]+ data Package = Package { pkgName :: Text, -- | relative to the monorepo, e.g. anda/devs/zed/stable pkgPath :: FilePath,- -- | Terra subrepo from anda.hcl labels, e.g. "extras"+ -- | from the anda.hcl labels pkgSubrepo :: Maybe Text,- -- | a `nightly` or `quaterdaily` label in anda.hcl: built from a branch- -- head on a schedule, not from releases+ -- | a `nightly` or `quaterdaily` label: built from a branch head, not releases pkgNightly :: Bool,- -- | the spec file, relative to the tree it was read from+ pkgUpdBranch :: Bool,+ -- | relative to the tree it was read from pkgSpecPath :: Maybe FilePath,- -- | hash of everything in the package dir that can change what rpmspec- -- says, plus the dist and macros. keys caches of anda runs+ -- | hash of everything that can change what rpmspec says, keying the anda+ -- run caches pkgFingerprint :: Text,- -- | what's merged in git pkgSpecEVR :: Maybe EVR,- -- | what's actually published pkgRepoBuild :: Maybe RepoBuild, pkgLastCommit :: Maybe UTCTime, -- | Nothing means no update.rhai pkgUpstream :: Maybe UpstreamSource,- -- | the update.rhai itself, for replaying it against the upstream value pkgRhai :: Maybe Text,- -- | only filled in when upstream lookups ran+ -- | only filled in with --upstream pkgUpstreamVer :: Maybe Text,- -- | why the upstream lookup failed, if it did pkgUpstreamErr :: Maybe Text,- -- | whether the repo this package publishes to was fetched. when it- -- wasn't, pkgRepoBuild = Nothing means "unknown", not "never published"+ pkgSubmoduleEvents :: [SubmoduleEvent],+ -- | when False, pkgRepoBuild = Nothing means "unknown", not "never published" pkgRepoChecked :: Bool } deriving stock (Eq, Show)@@ -127,19 +132,16 @@ data Finding = Finding { fCheck :: Text, fPackage :: Text,- -- | the package's directory in the monorepo. package names aren't unique- -- (see the duplicate-name check), so this is what identifies it+ -- | package names aren't unique, the monorepo path is fPath :: Maybe Text, fSeverity :: Severity, fMessage :: Text,- -- | the releases this showed up in, by name ("43", "rawhide"). empty- -- until the run stamps it+ -- | release names ("43", "rawhide"), empty until the run stamps them fReleases :: [Text] } deriving stock (Eq, Show, Generic) deriving anyclass (ToJSON) --- | a finding about a package finding :: Text -> Package -> Severity -> Text -> Finding finding check p sev msg = Finding@@ -152,44 +154,34 @@ } data Config = Config- { -- | local checkout of terrapkg/packages, or another anda monorepo- cfgMonorepo :: FilePath,- -- | the directory in it holding every package, e.g. anda+ { cfgMonorepo :: FilePath, cfgRoot :: FilePath,- -- | git ref to read specs from, e.g. origin/f43. Nothing reads the- -- working tree of cfgMonorepo as it is+ -- | Nothing reads the working tree as it is cfgSpecRef :: Maybe Text,- -- | the monorepo branch name, f43 or frawhide. update.rhai- -- scripts see it as labels.branch+ -- | update.rhai scripts see it as labels.branch cfgBranch :: Text,- -- | the main repo's name, e.g. terra44+ cfgRolling :: Bool, cfgRepoName :: Text,- -- | where it is: a metalink or a base url, with {repo} where the repo- -- name goes. without {repo} it's the main repo's url, and subrepos add- -- -extras to the end of it+ -- | a metalink or base url with {repo} where the repo name goes. without+ -- {repo} it's the main repo's url, and subrepos append -extras and so on cfgRepoUrl :: Text,- -- | repo base urls to check for overlap- cfgFedoraRepos :: [Text],- -- | Fedora/Terra release number, e.g. 44. rawhide has one too- cfgRelease :: Int,- -- | %{dist} used when evaluating specs, e.g. .fc44, .fcrawhide or .um44- cfgDist :: Text,+ cfgSubrepos :: Bool,+ cfgBaseRepos :: [Text],+ cfgBaseName :: Text,+ -- | Nothing undefines the macro+ cfgMacros :: [(Text, Maybe Text)],+ cfgCacheTag :: Text, cfgStaleAfter :: NominalDiffTime,- -- | whether to hit upstream APIs at all cfgUpstream :: Bool,- -- | reuse cached upstream answers younger than this cfgCacheFor :: NominalDiffTime,- -- | when this oglyad run started. upstream answers fetched since then are- -- reused whatever cfgCacheFor says, so checking four releases asks each- -- upstream once+ -- | upstream answers fetched since this are reused whatever cfgCacheFor+ -- says, so checking several releases asks each upstream once cfgRunStart :: UTCTime,- -- | parallel jobs, defaults to the number of cores cfgConcurrency :: Int, cfgGitHubToken :: Maybe Text } deriving stock (Show) --- | the url of the main repo, or of a subrepo like extras repoUrl :: Config -> Maybe Text -> Text repoUrl Config {..} sub | "{repo}" `T.isInfixOf` cfgRepoUrl = T.replace "{repo}" (cfgRepoName <> suffix) cfgRepoUrl@@ -197,18 +189,15 @@ where suffix = maybe "" ("-" <>) sub --- | everything a check is allowed to look at, already collected.--- checks never do IO, which is what keeps them trivially testable+-- | everything a check may look at. checks never do IO, which keeps them+-- trivially testable data Snapshot = Snapshot { -- | keyed by monorepo path, since names can repeat snapPackages :: Map FilePath Package,- -- | source package names present in Fedora- snapFedora :: Set Text,+ snapBase :: Set Text, snapNow :: UTCTime, snapConfig :: Config,- -- | the monorepo commit this run looked at snapHead :: Maybe Text,- -- | check names each package dir's anda.hcl asks to ignore snapIgnoreLabels :: Map FilePath [Text] }
src/Oglyad/Version.hs view
@@ -1,7 +1,5 @@--- | RPM version handling: EVR parsing and the rpmvercmp algorithm------ reference implementation: rpmio/rpmvercmp.c in rpm-software-management/rpm--- test vectors: tests/rpmvercmp.at in the same repo+-- | ports rpmio/rpmvercmp.c from rpm-software-management/rpm, tested+-- against its tests/rpmvercmp.at module Oglyad.Version ( EVR (..), parseEVR,@@ -15,19 +13,17 @@ import Data.Aeson (FromJSON, ToJSON) import Data.Char (isAlphaNum, isAsciiLower, isAsciiUpper, isDigit)-import Data.List (find, minimumBy)+import Data.List (minimumBy)+import Data.Maybe (fromMaybe) import Data.Ord (comparing) import Data.Text (Text) import Data.Text qualified as T import GHC.Generics (Generic) import Text.Read (readMaybe) --- | epoch:version-release------ note: 'Eq' is structural but 'Ord' goes through rpmvercmp, so "1.05" and--- "1.5" compare EQ while being /=. build-lag relies on the structural Eq on--- purpose, since any textual difference between git and the repo is worth--- reporting+-- | 'Eq' is structural but 'Ord' goes through rpmvercmp, so "1.05" and "1.5"+-- compare EQ while being /=. build-lag relies on that, since any textual+-- difference between git and the repo is worth reporting data EVR = EVR { evrEpoch :: !Int, evrVersion :: !Text,@@ -39,14 +35,13 @@ instance Ord EVR where compare = compareEVR --- | parse "[epoch:]version-release". a missing epoch or "(none)" means 0 parseEVR :: Text -> Maybe EVR parseEVR raw = do let (epochPart, rest) = case T.breakOn ":" raw of (vr, "") -> ("0", vr) (e, r) -> (e, T.drop 1 r) epoch <- if epochPart == "(none)" then Just 0 else readMaybe (T.unpack epochPart)- -- release is everything after the *last* dash, versions can't contain dashes+ -- versions can't contain dashes, so the release starts after the last one let (verDash, rel) = T.breakOnEnd "-" rest ver <- T.stripSuffix "-" verDash if T.null ver || T.null rel then Nothing else Just (EVR epoch ver rel)@@ -59,8 +54,7 @@ compareEVR (EVR e1 v1 r1) (EVR e2 v2 r2) = compare e1 e2 <> rpmvercmp v1 v2 <> rpmvercmp r1 r2 --- | compare two version (or release) strings exactly the way rpm does.--- this is a line-by-line port of rpmvercmp.c+-- | a line-by-line port of rpmvercmp.c -- -- * anything that isn't an ascii letter, digit, '~' or '^' is a separator -- * '~' sorts before everything, even the end of the string (1.0~rc1 < 1.0)@@ -81,17 +75,14 @@ let one = T.dropWhile isSep one0 two = T.dropWhile isSep two0 in case (T.uncons one, T.uncons two) of- -- tilde: whoever has it is older (Just ('~', one'), Just ('~', two')) -> go one' two' (Just ('~', _), _) -> LT (_, Just ('~', _)) -> GT- -- caret: older than anything except the end of the string (Just ('^', one'), Just ('^', two')) -> go one' two' (Nothing, Just ('^', _)) -> LT (Just ('^', _), Nothing) -> GT (Just ('^', _), Just _) -> LT (Just _, Just ('^', _)) -> GT- -- one side ran out (Nothing, Nothing) -> EQ (Just _, Nothing) -> GT (Nothing, Just _) -> LT@@ -100,8 +91,6 @@ | otherwise -> segment False (T.span isAlpha one) (T.span isAlpha two) segment isNum (seg1, rest1) (seg2, rest2)- -- the other side has a different kind of segment here.- -- numeric beats alpha | T.null seg2 = if isNum then GT else LT | isNum = let n1 = T.dropWhile (== '0') seg1@@ -122,22 +111,21 @@ -- so "6.0.0-alpha1" becomes "6.0.0~alpha1" normalizeVersion :: Text -> Text normalizeVersion raw =- let t = T.takeWhileEnd (/= '/') (T.strip raw)- noV = maybe t id (T.stripPrefix "v" t)- fromDigit = case T.dropWhile (not . isDigit) noV of- "" -> noV+ let afterLastSlash = T.takeWhileEnd (/= '/') (T.strip raw)+ withoutV = fromMaybe afterLastSlash (T.stripPrefix "v" afterLastSlash)+ fromFirstDigit = case T.dropWhile (not . isDigit) withoutV of+ "" -> withoutV d -> d- in T.replace "-" "." (tildePre fromDigit)+ in T.replace "-" "." (tildePrereleases fromFirstDigit) --- | "-rc1" / ".beta" / "_alpha2" → "~rc1" / "~beta" / "~alpha2"-tildePre :: Text -> Text-tildePre t = case T.uncons t of+tildePrereleases :: Text -> Text+tildePrereleases t = case T.uncons t of Nothing -> "" Just (c, rest)- | c `elem` ("-._" :: String), Just _ <- marker rest -> T.cons '~' (tildePre rest)- | otherwise -> T.cons c (tildePre rest)+ | c `elem` ("-._" :: String), startsWithPrerelease rest -> T.cons '~' (tildePrereleases rest)+ | otherwise -> T.cons c (tildePrereleases rest) where- marker r = find (`T.isPrefixOf` T.toLower r) ["alpha", "beta", "rc", "pre", "dev"]+ startsWithPrerelease r = any (`T.isPrefixOf` T.toLower r) ["alpha", "beta", "rc", "pre", "dev"] -- | compare an upstream version with a spec Version, forgiving the usual -- ways packagers respell a version. returns how upstream compares to the spec@@ -152,23 +140,20 @@ -- * otherwise rpmvercmp decides compareUpstream :: Text -> Text -> Ordering compareUpstream up spec- | squash u == squash s || squash u == squash (snapshotBase s) = EQ- | not (T.null (core u)) && core u == core s && not (isPre u || isPre s) = EQ+ | sameAlnums = EQ+ | sameLeadingNumber && not (isPrerelease u || isPrerelease s) = EQ | otherwise = rpmvercmp u s where u = normalizeVersion up s = normalizeVersion spec- squash = T.toLower . T.filter isAlphaNum- core = T.dropWhileEnd (== '.') . T.takeWhile (\c -> isDigit c || c == '.')- -- a prerelease is never "the same version with junk after it"- isPre v = "~" `T.isPrefixOf` T.drop (T.length (core v)) (T.dropWhileEnd (== '.') v)+ sameAlnums = alnums u `elem` [alnums s, alnums (snapshotBase s)]+ sameLeadingNumber = not (T.null (leadingNumber u)) && leadingNumber u == leadingNumber s+ alnums = T.toLower . T.filter isAlphaNum+ leadingNumber = T.dropWhileEnd (== '.') . T.takeWhile (\c -> isDigit c || c == '.')+ isPrerelease v = "~" `T.isPrefixOf` T.drop (T.length (leadingNumber v)) (T.dropWhileEnd (== '.') v) --- | a version with its snapshot suffix cut off: everything from a '^', or--- from a git marker like ".git" or "+git". not "~git", since that's a--- snapshot from before the release, so 2.0~git... really is behind 2.0------ >>> snapshotBase "2.0.0~alpha~1~20250926.gitfc62ae7"--- "2.0.0~alpha~1~20250926"+-- | "~git" isn't a snapshot marker, since 2.0~git... is a snapshot from+-- before the release and really is behind 2.0 snapshotBase :: Text -> Text snapshotBase v = let base = T.takeWhile (/= '^') v
test/Spec.hs view
@@ -17,16 +17,17 @@ import Oglyad.Check.DuplicateName qualified as DuplicateName import Oglyad.Check.FedoraOverlap qualified as FedoraOverlap import Oglyad.Check.Stale qualified as Stale+import Oglyad.Check.Submodules qualified as Submodules import Oglyad.Check.UnparsedRhai qualified as UnparsedRhai import Oglyad.Ignore import Oglyad.Process (childEnv, run) import Oglyad.Regex qualified as Re import Oglyad.Release import Oglyad.Report (ReleaseInfo (..), Report (..), renderJson, renderMarkdown, renderText)-import Oglyad.Source.Anda (scriptErrors, stripAnsi)-import Oglyad.Source.Monorepo (PkgDir (..), checkoutRef, findPackageDirs, hclString, lastCommits, macroFingerprint, specFingerprint, versionMacros)+import Oglyad.Source.Anda (ProjectRun (..), RunError (..), RunStatus (..), describeError, parseReport, stripAnsi)+import Oglyad.Source.Monorepo (PkgDir (..), checkoutRef, findPackageDirs, hclList, hclString, lastCommits, macroFingerprint, specFingerprint, versionMacros) import Oglyad.Source.Repodata (RpmEntry (..), decompressAndParse, fetchRepo, latestBySource, parseMetalink, parsePrimary, srpmName)-import Oglyad.Source.Rhai (parseUpdateRhai, parseUpdateRhaiWith)+import Oglyad.Source.Rhai (parseUpdateRhai, parseUpdateRhaiWith, tracksSubmodules) import Oglyad.Source.RhaiEval (Simulated (..), simulateVersion) import Oglyad.Source.Upstream (isSemver) import Oglyad.Types@@ -77,8 +78,8 @@ ("1.0^git1", "1.0^git2", LT), ("1.0^20160101", "1.0.1", LT) ]- it "is reflexive" $ property $ \(V a) -> rpmvercmp a a == EQ- it "is antisymmetric" $ property $ \(V a) (V b) ->+ it "is reflexive" $ property $ \(VersionLike a) -> rpmvercmp a a == EQ+ it "is antisymmetric" $ property $ \(VersionLike a) (VersionLike b) -> rpmvercmp a b == flipOrd (rpmvercmp b a) describe "normalizeVersion" $ do@@ -104,7 +105,6 @@ ("0.42.0", "0.41.0^20261031.9e2ed65", GT), -- a snapshot taken after the release it's based on isn't drift ("0.41.0", "0.41.0^20261031.9e2ed65", EQ),- -- snapshot suffixes don't stop a respelled version from matching ("2.0.0-alpha-1-20250926", "2.0.0~alpha~1~20250926.gitfc62ae7", EQ), ("0.4-2", "0.4~2^20260805git.d5d9f60", EQ), ("1.3", "1.2^20260101git.abc1234", GT),@@ -218,7 +218,6 @@ Map.keys (latestBySource entries) `shouldBe` ["zed"] it "rejects garbage" $ parsePrimary "<metadata><package>" `shouldSatisfy` isLeft it "decompresses zstd through the zstd command" $ do- -- the same fixture, compressed by the real zstd tool (_, compressed, _) <- readProcess (setStdin (byteStringInput primaryFixture) (proc "zstd" ["-c", "-q"])) r <- decompressAndParse "repodata/abc-primary.xml.zst" compressed fmap (map rpmName) r `shouldBe` Right ["zed", "zed-cli"]@@ -236,6 +235,9 @@ it "finds the spec" $ hclString "spec" hcl `shouldBe` Just "zed.spec" it "finds the subrepo" $ hclString "subrepo" hcl `shouldBe` Just "extras" it "doesn't match key prefixes" $ hclString "sub" hcl `shouldBe` Nothing+ it "reads lists like arches" $ do+ hclList "arches" "project pkg {\n arches = [\"aarch64\", \"x86_64\"]\n}\n" `shouldBe` ["aarch64", "x86_64"]+ hclList "arches" "project pkg {\n arch = \"x86_64\"\n}\n" `shouldBe` [] it "reads bare values like nightly = 1" $ hclString "nightly" "labels {\n nightly = 2 # slot\n}\n" `shouldBe` Just "2" describe "parseUpdateRhai" $ do@@ -278,23 +280,70 @@ it "doesn't guess NoVersion without the spec's Version: line" $ parseUpdateRhai "rpm.release();" `shouldSatisfy` isUnknown - describe "anda logs" $ do- it "pulls the reason out of a script exception" $- scriptErrors- " 2026-11-09T22:25:38.348511Z ERROR andax::run: Script Exception \8212\8212 lib/libfreeaptx\n\- \ \9500\9472\9552 Rhai: Syntax error: Expecting ';' to terminate this statement (line 5, position 39)\n\- \ in andax::run::traceback with proj=\"lib/libfreeaptx\"\n"- `shouldBe` ["Rhai: Syntax error: Expecting ';' to terminate this statement (line 5, position 39)"]- it "finds nothing in a clean log" $- scriptErrors " 2026-11-09T22:25:34Z DEBUG anda_config::config: Reading hcl file: \"anda.hcl\"\n" `shouldBe` []+ describe "anda reports" $ do+ it "reads a failed script" $+ parseReport+ "{\"version\": 1, \"projects\": [{\"project\": \"lib/x/pkg\", \"alias\": \"lib/x\", \"script\": \"anda/lib/x/update.rhai\", \"spec\": null, \"status\": \"failed\", \"spec_changed\": false, \"duration_ms\": 3, \"error\": {\"kind\": \"script\", \"message\": \"status code 404\", \"function\": \"gh\", \"line\": 3, \"column\": 9}}]}"+ `shouldBe` Right [ProjectRun "lib/x/pkg" Failed (Just (RunError "status code 404" (Just "gh") (Just 3))) []]+ it "reads a script that stopped itself" $+ parseReport "{\"version\": 1, \"projects\": [{\"project\": \"a\", \"status\": \"terminated\", \"error\": null}]}"+ `shouldBe` Right [ProjectRun "a" Terminated Nothing []]+ it "reads submodule events and skips kinds it doesn't know" $+ parseReport+ "{\"version\": 1, \"projects\": [{\"project\": \"a\", \"status\": \"ok\", \"error\": null, \"events\": [\+ \{\"kind\": \"submodule-added\", \"path\": \"lib/date\", \"url\": \"https://github.com/HowardHinnant/date\", \"commit\": \"abc\", \"global\": \"submodule_lib_date\"}, \+ \{\"kind\": \"submodule-dropped\", \"global\": \"submodule_lib_old\", \"commit\": \"def\"}, \+ \{\"kind\": \"something-new\"}]}]}"+ `shouldBe` Right+ [ ProjectRun+ "a"+ Ran+ Nothing+ [ SubmoduleAdded "lib/date" "https://github.com/HowardHinnant/date" "abc" "submodule_lib_date",+ SubmoduleDropped "submodule_lib_old" "def"+ ]+ ]+ it "refuses a newer report version" $+ parseReport "{\"version\": 2, \"projects\": []}" `shouldSatisfy` either (const True) (const False)+ it "names the failing helper and line" $ do+ describeError (RunError "status code 404" (Just "gh") (Just 3)) `shouldBe` "gh(): status code 404 (line 3)"+ describeError (RunError "upstream moved" Nothing Nothing) `shouldBe` "upstream moved"+ describeError (RunError "Function not found: header\nin call to function 'bodhi' (line 16)" Nothing (Just 7))+ `shouldBe` "Function not found: header, in call to function 'bodhi' (line 16) (line 7)" it "strips color codes" $ stripAnsi " 0: \ESC[91mProject not found: x\ESC[0m" `shouldBe` " 0: Project not found: x" - describe "unparsed-rhai with anda" $+ describe "submodules" $ do+ it "finds rpm.submodules() calls" $ do+ tracksSubmodules "let tag = gh(\"a/b\");\nrpm.version(tag);\nrpm.submodules(gh_submodules(\"a/b\", tag));" `shouldBe` True+ tracksSubmodules "// rpm.submodules(x)\nrpm.version(gh(\"a/b\"));" `shouldBe` False+ it "reports added and dropped submodules" $ do+ let found =+ Submodules.check+ snapshot+ pkg+ { pkgSubmoduleEvents =+ [ SubmoduleAdded "lib/date" "https://github.com/HowardHinnant/date" (T.replicate 40 "a") "submodule_lib_date",+ SubmoduleDropped "submodule_lib_old" (T.replicate 40 "b")+ ]+ }+ map (\f -> (fCheck f, fSeverity f)) found `shouldBe` [("submodules", Warn), ("submodules", Warn)]+ map fMessage found+ `shouldBe` [ "upstream added the submodule lib/date (https://github.com/HowardHinnant/date at aaaaaaaaaaaa). add %global submodule_lib_date with a Source and %submodule_setup line, or pass it to rpm.submodules() as ignore",+ "upstream dropped the submodule for %global submodule_lib_old. remove the global, its Source and its %submodule_setup line"+ ]+ it "is quiet without events" $+ Submodules.check snapshot pkg `shouldBe` []++ describe "unparsed-rhai with anda" $ do+ let unknown = pkg {pkgUpstream = Just (Unknown "let x = get(\"u\");"), pkgUpdBranch = True} it "stays quiet once anda answered for the script" $ do- let unknown = pkg {pkgUpstream = Just (Unknown "let x = get(\"u\");")} map fCheck (UnparsedRhai.check snapshot unknown) `shouldBe` ["unparsed-rhai"] UnparsedRhai.check snapshot unknown {pkgUpstreamVer = Just "1.3.0"} `shouldBe` []+ it "only reports scripts without updbranch on rawhide" $ do+ let rawhide = snapshot {snapConfig = (snapConfig snapshot) {cfgRolling = True}}+ UnparsedRhai.check snapshot unknown {pkgUpdBranch = False} `shouldBe` []+ map fCheck (UnparsedRhai.check rawhide unknown {pkgUpdBranch = False}) `shouldBe` ["unparsed-rhai"] describe "versionMacros" $ do it "finds %name, %{name} and %{?name}" $@@ -331,9 +380,12 @@ $ severities (UnparsedRhai.check snapshot (pkg {pkgRhai = Just "let v = gh_tag(\"a/b\"));\nrpm.version(v);"})) `shouldBe` [Warn] describe "fedora-overlap check" $ do- let withFedora = snapshot {snapFedora = Set.singleton "zed"}+ let withFedora = snapshot {snapBase = Set.singleton "zed"} it "warns when Fedora ships the same package" $ severities (FedoraOverlap.check withFedora pkg) `shouldBe` [Warn]+ it "names the base distro it overlaps with" $ do+ let suse = withFedora {snapConfig = (snapConfig withFedora) {cfgBaseName = "openSUSE"}}+ map fMessage (FedoraOverlap.check suse pkg) `shouldBe` ["also shipped by openSUSE"] it "ignores packages in subrepos" $ FedoraOverlap.check withFedora (pkg {pkgSubrepo = Just "multimedia"}) `shouldBe` [] it "is quiet when Fedora doesn't have it" $@@ -428,7 +480,7 @@ createDirectoryIfMissing True (root </> "anda/a") writeFile (root </> "anda/a/a.spec") "Name: a" createFileLink "does-not-exist" (root </> "anda/a/broken")- key <- specFingerprint "m" ".fc44" root "anda/a"+ key <- specFingerprint "m" [("dist", Just ".fc44")] root "anda/a" T.length key `shouldBe` 64 it "doesn't follow a symlink loop" $ withSystemTempDirectory "oglyad-loop" $ \root -> do@@ -437,9 +489,17 @@ createDirectoryLink ".." (root </> "anda/a/loop") dirs <- findPackageDirs root "anda" map (fmap pdPath) dirs `shouldBe` [Right "anda/a"]- _ <- specFingerprint "m" ".fc44" root "anda/a"+ _ <- specFingerprint "m" [("dist", Just ".fc44")] root "anda/a" pure () + it "walks packages at the top of the checkout, skipping the monorepo's own anda.hcl" $+ withSystemTempDirectory "oglyad-top" $ \root -> do+ writeFile (root </> "anda.hcl") "config {}\nproject {}"+ createDirectoryIfMissing True (root </> "devs/a")+ writeFile (root </> "devs/a/anda.hcl") "project pkg {}"+ dirs <- findPackageDirs root ""+ map (fmap pdPath) dirs `shouldBe` [Right "devs/a"]+ describe "nightly labels" $ it "treats nightly and quaterdaily as branch-head builds, not weekly" $ withSystemTempDirectory "oglyad-labels" $ \root -> do@@ -484,32 +544,80 @@ fpAfter `shouldNotBe` fpBefore describe "releases" $ do- it "parses --release spellings" $- map parseReleaseArg ["43", "f44", "rawhide", "frawhide", "el10", ""]- `shouldBe` [Right (NumberedArg 43), Right (NumberedArg 44), Right RawhideArg, Right RawhideArg, Left "unknown release \"el10\", expected a number like 43, f43, or rawhide", Left "unknown release \"\", expected a number like 43, f43, or rawhide"]+ it "parses --release spellings" $ do+ let unknown raw = Left ("unknown release " <> show (raw :: String) <> ", expected a number like 43, f43, rawhide, tumbleweed, or el10")+ map parseReleaseArg ["43", "f44", "rawhide", "frawhide", "tumbleweed", "tw", "el10", "el", "epel10", ""]+ `shouldBe` [Right (NumberedArg 43), Right (NumberedArg 44), Right RawhideArg, Right RawhideArg, Right TumbleweedArg, Right TumbleweedArg, Right (ElArg 10), unknown "el", unknown "epel10", unknown ""] it "spells rawhide's dist, branch and repos the way Terra does" $ do- let raw = Release "rawhide" 46+ let raw = Release Fedora "rawhide" 46 (releaseDist terra raw, releaseBranch terra raw, repoName terra raw, releaseLabel raw) `shouldBe` (".fcrawhide", "frawhide", "terrarawhide", "rawhide")- fedoraReposFor raw `shouldBe` ["https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os"]- length (fedoraReposFor (Release "43" 43)) `shouldBe` 2+ baseReposFor raw `shouldBe` ["https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os"]+ length (baseReposFor (Release Fedora "43" 43)) `shouldBe` 2+ releaseMacros terra raw `shouldBe` [("dist", Just ".fcrawhide"), ("fedora", Just "46")] it "spells releases the way Ultramarine does" $ do- let r44 = Release "44" 44+ let r44 = Release Fedora "44" 44 (releaseDist ultramarine r44, releaseBranch ultramarine r44, repoName ultramarine r44) `shouldBe` (".um44", "um44", "um44") lookupDistro "ultramarine" `shouldBe` Right ultramarine lookupDistro "umbrella" `shouldSatisfy` either (const True) (const False)+ it "spells tumbleweed the way Tuatara does" $ do+ (releaseDist tuatara tumbleweed, releaseBranch tuatara tumbleweed, repoName tuatara tumbleweed, releaseLabel tumbleweed)+ `shouldBe` ("", "tumbleweed", "tuatara-tumbleweed", "tumbleweed")+ isRolling tumbleweed `shouldBe` True+ releaseMacros tuatara tumbleweed `shouldBe` [("dist", Nothing), ("fedora", Nothing), ("suse_version", Just "1699"), ("is_opensuse", Just "1")]+ repoUrlFor (distroRepoUrl tuatara) tumbleweed `shouldBe` "https://repos.fyralabs.com/{repo}"+ baseReposFor tumbleweed `shouldBe` ["https://download.opensuse.org/tumbleweed/repo/oss", "https://download.opensuse.org/update/tumbleweed"]+ lookupDistro "tuatara" `shouldBe` Right tuatara+ it "spells el10 the way Terra EL does" $ do+ let el10 = el 10+ (releaseDist terra el10, releaseBranch terra el10, repoName terra el10, releaseLabel el10)+ `shouldBe` (".el10", "el10", "terrael10", "el10")+ isRolling el10 `shouldBe` False+ releaseSubrepos terra el10 `shouldBe` False+ releaseSubrepos terra (Release Fedora "44" 44) `shouldBe` True+ releaseMacros terra el10 `shouldBe` [("dist", Just ".el10"), ("fedora", Nothing), ("rhel", Just "10"), ("el10", Just "1"), ("epel", Just "10")]+ baseReposFor el10+ `shouldBe` [ "https://repo.almalinux.org/almalinux/10/BaseOS/x86_64/os",+ "https://repo.almalinux.org/almalinux/10/AppStream/x86_64/os",+ "https://repo.almalinux.org/almalinux/10/CRB/x86_64/os",+ "https://repo.almalinux.org/almalinux/10/extras/x86_64/os",+ "https://dl.fedoraproject.org/pub/epel/10/Everything/x86_64"+ ]+ releaseCacheTag terra el10 `shouldNotBe` releaseCacheTag terra (Release Fedora "10" 10)+ it "spells packages-el's releases the same way, minus Fedora" $ do+ let el10 = el 10+ (releaseDist terraEl el10, releaseBranch terraEl el10, repoName terraEl el10, releaseSubrepos terraEl el10)+ `shouldBe` (".el10", "el10", "terrael10", False)+ distroBases terraEl `shouldBe` [EL]+ lookupDistro "terra-el" `shouldBe` Right terraEl+ releaseCacheTag terraEl el10 `shouldNotBe` releaseCacheTag terra el10+ it "finds EL releases among a remote's branches" $+ elReleasesFrom ["HEAD", "el11", "f44", "el10", "el", "elx", "el10", "cypress/el10"]+ `shouldBe` [el 10, el 11]+ it "reads the repo name out of remote urls" $+ map+ remoteRepoName+ [ "git@github.com:terrapkg/packages-el.git",+ "https://github.com/terrapkg/packages-el/",+ "git@github.com:terrapkg/packages",+ "/srv/git/tuatara.git",+ "packages"+ ]+ `shouldBe` [Just "packages-el", Just "packages-el", Just "packages", Just "tuatara", Nothing]+ it "keeps cache tags apart across distros" $+ releaseCacheTag terra (Release Fedora "rawhide" 46) `shouldNotBe` releaseCacheTag ultramarine (Release Fedora "rawhide" 46) it "fills in repo url templates, subrepos included" $ do- let cfg url = (snapConfig snapshot) {cfgRepoName = "terra44", cfgRepoUrl = repoUrlFor url (Release "44" 44)}+ let cfg url = (snapConfig snapshot) {cfgRepoName = "terra44", cfgRepoUrl = repoUrlFor url (Release Fedora "44" 44)} repoUrl (cfg defaultRepoUrl) (Just "extras") `shouldBe` "https://tetsudou.fyralabs.com/metalink?repo=terra44-extras&arch=x86_64" repoUrl (cfg "https://example.com/um{release}/x86_64") Nothing `shouldBe` "https://example.com/um44/x86_64" repoUrl (cfg "https://example.com/terra44") (Just "extras") `shouldBe` "https://example.com/terra44-extras" it "takes Fedora releases from Bodhi, rawhide last" $- parseBodhiReleases bodhiFixture `shouldBe` Right [Release "43" 43, Release "44" 44, Release "45" 45, Release "rawhide" 46]+ parseBodhiReleases bodhiFixture `shouldBe` Right [Release Fedora "43" 43, Release Fedora "44" 44, Release Fedora "45" 45, Release Fedora "rawhide" 46] it "complains when Bodhi lists nothing useful" $ parseBodhiReleases "{\"releases\": []}" `shouldSatisfy` either (const True) (const False) describe "mergeFindings" $ do- let r43 = Release "43" 43- r44 = Release "44" 44+ let r43 = Release Fedora "43" 43+ r44 = Release Fedora "44" 44 lag dist = Finding "build-lag" "zed" (Just "anda/zed") Error ("never published (spec is at 1.2-1" <> dist <> ")") [] drift = Finding "drift" "zed" (Just "anda/zed") Warn "upstream has 1.3, spec is at 1.2" [] it "merges findings that only differ by dist" $@@ -518,6 +626,12 @@ it "merges by the distro's own dist" $ mergeFindings ultramarine [(r43, [lag ".um43"]), (r44, [lag ".um44"])] `shouldBe` [(lag "%{dist}") {fReleases = ["43", "44"]}]+ it "leaves messages alone when the distro has no dist tag" $+ mergeFindings tuatara [(tumbleweed, [lag "", drift])]+ `shouldBe` [(lag "") {fReleases = ["tumbleweed"]}, drift {fReleases = ["tumbleweed"]}]+ it "merges EL findings by their own dist" $+ mergeFindings terra [(el 9, [lag ".el9"]), (el 10, [lag ".el10"])]+ `shouldBe` [(lag "%{dist}") {fReleases = ["el9", "el10"]}] it "keeps the real message for a single release" $ mergeFindings terra [(r43, [drift]), (r44, [lag ".fc44"])] `shouldBe` [drift {fReleases = ["43"]}, (lag ".fc44") {fReleases = ["44"]}]@@ -613,8 +727,6 @@ severities :: [Finding] -> [Severity] severities = map fSeverity --- * fixtures- now :: UTCTime now = UTCTime (fromGregorian 2026 11 9) 0 @@ -628,6 +740,7 @@ pkgPath = "anda/devs/zed/stable", pkgSubrepo = Nothing, pkgNightly = False,+ pkgUpdBranch = False, pkgSpecPath = Just "anda/devs/zed/stable/zed.spec", pkgFingerprint = "", pkgSpecEVR = Just (EVR 0 "1.2.0" "1.fc44"),@@ -637,6 +750,7 @@ pkgRhai = Nothing, pkgUpstreamVer = Nothing, pkgUpstreamErr = Nothing,+ pkgSubmoduleEvents = [], pkgRepoChecked = True } @@ -644,7 +758,7 @@ snapshot = Snapshot { snapPackages = Map.singleton (pkgPath pkg) pkg,- snapFedora = Set.empty,+ snapBase = Set.empty, snapNow = now, snapConfig = Config@@ -652,11 +766,14 @@ cfgRoot = "anda", cfgSpecRef = Nothing, cfgBranch = "f44",+ cfgRolling = False, cfgRepoName = "terra44", cfgRepoUrl = defaultRepoUrl,- cfgFedoraRepos = [],- cfgRelease = 44,- cfgDist = ".fc44",+ cfgSubrepos = True,+ cfgBaseRepos = [],+ cfgBaseName = "Fedora",+ cfgMacros = [("dist", Just ".fc44"), ("fedora", Just "44")],+ cfgCacheTag = "-terra-44", cfgStaleAfter = 180 * nominalDay, cfgUpstream = False, cfgCacheFor = 0,@@ -668,7 +785,7 @@ snapIgnoreLabels = Map.empty } --- | trimmed from tetsudou's answer for terra44, with a second mirror+-- trimmed from tetsudou's answer for terra44, with a second mirror metalinkFixture :: LBS8.ByteString metalinkFixture = "<?xml version=\"1.0\" encoding=\"utf-8\"?><metalink version=\"3.0\" xmlns=\"http://www.metalinker.org/\" xmlns:mm0=\"http://fedorahosted.org/mirrormanager\" type=\"dynamic\" generator=\"tetsudou\"><files><file name=\"repomd.xml\">\@@ -727,14 +844,12 @@ \</package>\n\ \</metadata>\n" --- | version-ish strings: digits, letters and the separators rpm cares about-newtype V = V Text deriving stock (Show)+newtype VersionLike = VersionLike Text deriving stock (Show) -instance Arbitrary V where- arbitrary = V . T.pack <$> listOf (elements "0123456789abz.~^_")- shrink (V t) = V . T.pack <$> shrink (T.unpack t)+instance Arbitrary VersionLike where+ arbitrary = VersionLike . T.pack <$> listOf (elements "0123456789abz.~^_")+ shrink (VersionLike t) = VersionLike . T.pack <$> shrink (T.unpack t) --- | set an environment variable for the duration of an action withEnv :: String -> String -> IO a -> IO a withEnv key val act = bracket