diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,29 @@ ## Unreleased +## 0.3.1.0 + +### Bug Fixes + +- Export `CacheKey (..)`, `getCacheKey`, `CowCapability (..)`, and + `CowMethod (..)` from `EphemeralPg`. `clearCache` takes a `CacheKey` and + `CacheConfig.cow` holds a `Maybe CowCapability`, but neither type was reachable + from any exposed module, so `clearCache` could not be called and `cow` could + not be set to anything but `Nothing`. +- Export `withCachedConfig` from `EphemeralPg`. It was the only bracketing entry + point that accepts both a `Config` and a `CacheConfig`, so cached callers had no + way to set `temporaryRoot` without reimplementing `startCached`/`stop` bracketing. + +### Other Changes + +- Document that the startup sweep is scoped to a single temporary root, and that + an unset `temporaryRoot` resolves to `$TMPDIR`. Environments that allocate a + per-session `$TMPDIR` (`nix develop`, `nix-shell`, systemd `PrivateTmp`, some CI + runners) sweep a fresh empty directory on every run and never reclaim clusters + abandoned by earlier sessions. See `docs/temporary-roots-and-stale-cleanup.md`. +- Ship `docs/temporary-roots-and-stale-cleanup.md` in the source distribution via + `extra-doc-files`, and correct the `withCachedConfig` haddock example. + ## 0.3.0.0 ### Breaking Changes diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -516,7 +516,17 @@ ``` Create a custom temporary root before using it. The sweep returns sorted canonical -paths actually removed. Disable automatic sweeping with +paths actually removed. + +The sweep only inspects its own temporary root. With `temporaryRoot` unset that +root is `$TMPDIR`, which `nix develop`, `nix-shell`, systemd `PrivateTmp` and some +CI runners make unique per session — each run then sweeps a fresh empty directory +and never reclaims what earlier runs abandoned. Set a stable `temporaryRoot` and +use `withCachedConfig` or `withConfig` (the zero-argument `withCached` and `with` +always resolve to `$TMPDIR`). See +[Temporary roots and stale cleanup](docs/temporary-roots-and-stale-cleanup.md). + +Disable automatic sweeping with `config { Pg.sweepStaleOnStart = Last (Just False) }`; explicit sweeps still run, and new temporary instances still hold ownership locks. An absent setting enables automatic sweeping. diff --git a/docs/temporary-roots-and-stale-cleanup.md b/docs/temporary-roots-and-stale-cleanup.md new file mode 100644 --- /dev/null +++ b/docs/temporary-roots-and-stale-cleanup.md @@ -0,0 +1,127 @@ +# Temporary roots and stale cleanup + +`start` and `startCached` sweep abandoned PostgreSQL clusters before allocating a +new one, so a consumer killed with `SIGKILL` does not leak a postmaster forever. +The sweep is scoped to a single directory: the configured `temporaryRoot`, or the +system temporary directory when that is unset. + +That scope is the part worth understanding. If the temporary root changes between +runs, the sweep never sees what earlier runs left behind, and abandoned clusters +accumulate indefinitely while the feature appears to be enabled. + +## The default is `$TMPDIR`, which is not always stable + +With `temporaryRoot` unset, the root comes from +`System.Directory.getTemporaryDirectory`, which returns `$TMPDIR` on POSIX and +falls back to `/tmp`. Several common environments give each session its own +`$TMPDIR`: + +| Environment | `$TMPDIR` | +| --- | --- | +| `nix develop` / `nix-shell` | `/tmp/nix-shell.XXXXXX`, new per shell | +| systemd units with `PrivateTmp=yes` | private `/tmp` per service start | +| Some CI runners and container images | per-job or per-step directory | +| macOS launchd services | per-user `/var/folders/...` | + +Under any of these, a test run allocates its clusters inside that session's +directory. When the run is killed, the postmaster survives and keeps that +directory alive. The next run gets a *different* `$TMPDIR`, sweeps an empty new +root, finds nothing, and leaks again. + +The orphans are still perfectly reapable — they simply are not being looked at. +Nothing in the safety protocol rejects them. + +### What this looks like + +Four clusters abandoned by four separate `nix develop` sessions, each in its own +root, still running well over a day later: + +``` +PID 44741 /tmp/nix-shell.Ex1kKN/ephpg-data--81268893861081c0 port 62643 +PID 45523 /tmp/nix-shell.Vwr5fv/ephpg-data--67042a9bec14c9a7 port 62675 +PID 48818 /tmp/nix-shell.mofVo9/ephpg-data--6ae20a3606256f38 port 62710 +PID 58308 /tmp/nix-shell.pPgbfp/ephpg-data--553fa4c95bdb49fe port 62872 +``` + +Every one had been reparented to PID 1 and held zero client connections — that is, +they satisfied the identity requirements a sweep imposes on a live untracked +postmaster. They survived only because no later session ever searched those roots. + +## Set a stable root + +Point `temporaryRoot` at a path that does not change between runs, and create it +before use: + +```haskell +import Data.Monoid (Last (..)) +import EphemeralPg qualified as Pg +import System.Directory (createDirectoryIfMissing) + +testConfig :: IO Pg.Config +testConfig = do + let root = "/tmp/ephpg-my-project" + createDirectoryIfMissing True root + pure Pg.defaultConfig { Pg.temporaryRoot = Last (Just root) } +``` + +Then use the config-taking entry point rather than the zero-argument convenience +wrapper: + +```haskell +config <- testConfig +result <- Pg.withCachedConfig config Pg.defaultCacheConfig $ \db -> + -- Use the database... +``` + +`withCached` and `with` take no `Config`, so they always resolve to `$TMPDIR`. +Reach for `withCachedConfig` or `withConfig` as soon as you need a stable root. + +### Choosing the path + +- **Keep it short.** The socket directory lives under the same root, and + `validateSocketPath` rejects anything that would push the Unix socket path past + the platform limit. A deep root fails at startup rather than silently. +- **Keep it per-user on shared machines.** The instance registry is already + namespaced as `.ephemeral-pg-instances-`, but the root directory itself is + not. Include the user name if several accounts share the host. +- **Do not point it at a permanent data directory.** Permanent data, sockets, + snapshots and caches are excluded from sweeping by design. + +## Sharing one root across suites + +A shared root is safe for concurrent runs and is the intended arrangement. Live +instances hold exclusive lifetime locks, and a sweep claims only unlocked +candidates without blocking, so a sweep in one suite cannot reap a cluster another +suite is actively using. Multiple packages in the same project can and should +point at the same root — that is what lets a run of one suite clean up after a +killed run of another. + +## Verifying it works + +Cleanup after an abnormal exit is deferred to the *next* startup, so a sweep is +observable only on a subsequent run. To check the wiring end to end: + +1. Start an instance, then `SIGKILL` the consumer process (not the postmaster). +2. Confirm the postmaster survives and its data directory is still present under + the root. +3. Start a second instance from a **new shell session**. +4. Confirm the abandoned postmaster is gone and the directory was removed. + +Step 3 is the one that matters: running it from the same shell passes even with a +per-session `$TMPDIR`, which is exactly the bug this document is about. + +`sweepStaleInstances` performs the same sweep on demand and returns the sorted +canonical paths it actually removed, which makes it convenient as an assertion: + +```haskell +removed <- Pg.sweepStaleInstances config +``` + +It honours `temporaryRoot` independently of `sweepStaleOnStart`, so it still works +when automatic sweeping is disabled. + +## Related + +- [ADR 1: Stale instance ownership and reaping](adr/1-stale-instance-ownership-and-reaping.md) + — the ownership, identity and signalling protocol. +- `README.md`, "Cleanup after a killed consumer" — the short version. diff --git a/ephemeral-pg.cabal b/ephemeral-pg.cabal --- a/ephemeral-pg.cabal +++ b/ephemeral-pg.cabal @@ -1,6 +1,6 @@ cabal-version: 3.0 name: ephemeral-pg -version: 0.3.0.0 +version: 0.3.1.0 synopsis: Temporary PostgreSQL databases for testing description: A modern library for creating temporary PostgreSQL instances for testing. @@ -15,6 +15,7 @@ extra-doc-files: CHANGELOG.md README.md + docs/temporary-roots-and-stale-cleanup.md source-repository head type: git diff --git a/src/EphemeralPg.hs b/src/EphemeralPg.hs --- a/src/EphemeralPg.hs +++ b/src/EphemeralPg.hs @@ -43,6 +43,7 @@ with, withConfig, withCached, + withCachedConfig, start, startCached, sweepStaleInstances, @@ -59,7 +60,11 @@ -- * Cache Management CacheConfig (..), + CacheKey (..), + CowCapability (..), + CowMethod (..), defaultCacheConfig, + getCacheKey, clearCache, clearAllCaches, @@ -103,6 +108,7 @@ ) import EphemeralPg.Internal.Cache ( CacheConfig (..), + CacheKey (..), cleanupRuntimeFiles, clearAllCaches, clearCache, @@ -112,6 +118,10 @@ isCached, restoreFromCache, ) +import EphemeralPg.Internal.CopyOnWrite + ( CowCapability (..), + CowMethod (..), + ) import EphemeralPg.Internal.Directory ( createTempDataDirectory, createTempSocketDirectory, @@ -380,6 +390,18 @@ withCached = withCachedConfig defaultConfig defaultCacheConfig -- | Like 'withCached' but with custom configuration. +-- +-- Set 'temporaryRoot' here when the caller runs under a per-session @TMPDIR@ +-- (@nix develop@, @nix-shell@, systemd @PrivateTmp@, some CI runners). The +-- startup sweep only inspects its own temporary root, so a per-session root +-- hides clusters abandoned by earlier sessions. See +-- @docs/temporary-roots-and-stale-cleanup.md@. +-- +-- @ +-- let config = 'defaultConfig' { temporaryRoot = Last (Just root) } +-- 'withCachedConfig' config 'defaultCacheConfig' $ \\db -> do +-- -- Use the database... +-- @ withCachedConfig :: Config -> CacheConfig -> (Database -> IO a) -> IO (Either StartError a) withCachedConfig config cacheConfig action = mask $ \restore -> runStartup $ do db <- liftE $ startCached config cacheConfig