# ephemeral-pg
A modern Haskell library for creating temporary PostgreSQL databases for testing.
## Features
- Native hasql integration
- initdb caching for fast startup
- Copy-on-write support (macOS/Linux)
- Filesystem snapshots
- No shell injection vulnerabilities
- Type-safe configuration
## Requirements
- GHC 9.6+
- PostgreSQL 14+
## Installation
Add to your `cabal` file:
```cabal
build-depends:
ephemeral-pg
```
Or with Stack, add to your `package.yaml`:
```yaml
dependencies:
- ephemeral-pg
```
## Quick Start
```haskell
import EphemeralPg qualified as Pg
import Hasql.Connection qualified as Connection
main :: IO ()
main = do
result <- Pg.with \db -> do
Right conn <- Connection.acquire (Pg.connectionSettings db)
-- Use the connection...
Connection.release conn
case result of
Left err -> putStrLn $ "Error: " <> Pg.renderStartError err
Right () -> putStrLn "Success!"
```
## Usage Examples
### Basic Usage
The simplest way to use ephemeral-pg is with the `with` function, which creates
a temporary database, runs your action, and cleans up automatically:
```haskell
import EphemeralPg qualified as Pg
import Hasql.Connection qualified as Connection
import Hasql.Session qualified as Session
import Hasql.Statement qualified as Statement
testQuery :: IO ()
testQuery = do
result <- Pg.with \db -> do
Right conn <- Connection.acquire (Pg.connectionSettings db)
-- Run a simple query
result <- Session.run (Session.statement () selectOne) conn
Connection.release conn
pure result
case result of
Left err -> putStrLn $ "Startup error: " <> Pg.renderStartError err
Right (Left sessionErr) -> putStrLn $ "Query error: " <> show sessionErr
Right (Right value) -> putStrLn $ "Result: " <> show value
where
selectOne = Statement.Statement "SELECT 1" mempty decoder True
decoder = Decoders.singleRow (Decoders.column (Decoders.nonNullable Decoders.int4))
```
### Custom Configuration
You can customize the database configuration:
```haskell
import EphemeralPg qualified as Pg
import EphemeralPg.Config qualified as Config
customTest :: IO ()
customTest = do
let config = Pg.defaultConfig
{ Config.databaseName = "mytest"
, Config.postgresSettings =
[ ("log_statement", "'all'")
, ("log_min_duration_statement", "0")
]
}
result <- Pg.withConfig config \db -> do
-- Database name is "mytest"
-- All statements are logged
pure ()
case result of
Left err -> putStrLn $ Pg.renderStartError err
Right () -> putStrLn "Done!"
```
### Using Verbose Configuration
For debugging, use the pre-configured verbose settings:
```haskell
import EphemeralPg qualified as Pg
debugTest :: IO ()
debugTest = do
result <- Pg.withConfig Pg.verboseConfig \db -> do
-- All statements logged with duration
pure ()
pure ()
```
### Using auto_explain
For query plan analysis:
```haskell
import EphemeralPg qualified as Pg
analyzeQueries :: IO ()
analyzeQueries = do
result <- Pg.withConfig Pg.autoExplainConfig \db -> do
-- Query plans automatically logged for slow queries
pure ()
pure ()
```
### Caching for Fast Startup
For faster test suite execution, use caching. The first run initializes the
cache, and subsequent runs copy from it (using CoW if available):
```haskell
import EphemeralPg qualified as Pg
cachedTest :: IO ()
cachedTest = do
-- First call: ~2s (runs initdb and caches result)
-- Subsequent calls: ~200ms (copies from cache)
result <- Pg.withCached \db -> do
-- Use the database...
pure ()
pure ()
```
### Suite-level fixtures with template databases
For larger integration suites where many examples need the same migrated
schema, consider starting one cached PostgreSQL server for the whole suite,
migrating a template database once, and cloning clean per-example databases
with PostgreSQL's `CREATE DATABASE ... TEMPLATE ...`.
See [Suite-level template databases](docs/suite-template-databases.md) for the
full pattern, tradeoffs, and a complete fixture implementation.
### Manual Lifecycle Management
For more control, use `start` and `stop` directly:
```haskell
import EphemeralPg qualified as Pg
import Control.Exception (bracket)
manualLifecycle :: IO ()
manualLifecycle = do
result <- Pg.start Pg.defaultConfig
case result of
Left err -> putStrLn $ Pg.renderStartError err
Right db -> do
-- Use the database...
putStrLn $ "Database running on port: " <> show db.port
-- Clean up when done
Pg.stop db
```
Or with bracket for exception safety:
```haskell
import EphemeralPg qualified as Pg
import Control.Exception (bracket)
bracketExample :: IO ()
bracketExample = do
result <- Pg.start Pg.defaultConfig
case result of
Left err -> putStrLn $ Pg.renderStartError err
Right db ->
bracket (pure db) Pg.stop \db' -> do
-- Use the database...
pure ()
```
### Restarting the Database
You can restart the database to apply configuration changes or test recovery:
```haskell
import EphemeralPg qualified as Pg
restartTest :: IO ()
restartTest = do
result <- Pg.with \db -> do
let port1 = db.port
-- Restart the server
restartResult <- Pg.restart db
case restartResult of
Left err -> fail $ "Restart failed: " <> show err
Right db' -> do
-- Server restarted, data preserved
let port2 = db'.port
-- Port remains the same
pure (port1 == port2)
case result of
Left err -> putStrLn $ Pg.renderStartError err
Right same -> putStrLn $ "Ports same: " <> show same
```
### Snapshots
Create filesystem snapshots for fast test isolation:
```haskell
import EphemeralPg qualified as Pg
import EphemeralPg.Snapshot qualified as Snapshot
snapshotTest :: IO ()
snapshotTest = do
result <- Pg.with \db -> do
-- Set up initial state
-- ... create tables, insert data ...
-- Create a snapshot
Right snapshot <- Snapshot.createSnapshot db
-- Run a destructive test
-- ... delete everything ...
-- Restore to the snapshot
Right () <- Snapshot.restoreSnapshot snapshot db
-- Data is restored
-- Clean up snapshot when done
Snapshot.deleteSnapshot snapshot
pure ()
pure ()
```
### Dump and Restore
Export and import database contents:
```haskell
import EphemeralPg qualified as Pg
import EphemeralPg.Dump qualified as Dump
dumpTest :: IO ()
dumpTest = do
result <- Pg.with \db -> do
-- Set up data
-- ...
-- Dump to file
Right () <- Dump.dump db "/tmp/backup.sql"
pure ()
-- Later, restore to a new database
result2 <- Pg.with \db2 -> do
Right () <- Dump.restore db2 "/tmp/backup.sql"
-- Data is now in db2
pure ()
pure ()
```
## Configuration Reference
### Config Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `databaseName` | `Text` | `"postgres"` | Name of the database to create |
| `user` | `Text` | Current user | PostgreSQL username |
| `password` | `Maybe Text` | `Nothing` | PostgreSQL password (`Nothing` = trust auth) |
| `port` | `Last Word16` | Auto-assigned | Port number (finds free port if not specified) |
| `dataDirectory` | `DirectoryConfig` | Temporary | Data directory location |
| `socketDirectory` | `DirectoryConfig` | Temporary | Unix socket directory |
| `temporaryRoot` | `Last FilePath` | System temp | Root for temporary directories |
| `postgresSettings` | `[(Text, Text)]` | Optimized defaults | postgresql.conf settings |
| `initDbArgs` | `[Text]` | `[]` | Additional initdb arguments |
### Pre-configured Configs
- `defaultConfig` - Basic configuration with optimized defaults
- `verboseConfig` - All statements logged with duration
- `autoExplainConfig` - Query plans logged for slow queries
### PostgreSQL Settings Defaults
The default configuration includes these optimized settings for testing:
```
shared_buffers = 12MB
fsync = off
synchronous_commit = off
full_page_writes = off
log_min_duration_statement = 0
log_connections = on
log_disconnections = on
random_page_cost = 1.0
```
## API Reference
### Core Functions
```haskell
-- Bracket-style (recommended)
with :: (Database -> IO a) -> IO (Either StartError a)
withConfig :: Config -> (Database -> IO a) -> IO (Either StartError a)
-- With caching
withCached :: (Database -> IO a) -> IO (Either StartError a)
-- Manual lifecycle
start :: Config -> IO (Either StartError Database)
stop :: Database -> IO ()
restart :: Database -> IO (Either StartError Database)
```
### Database Handle
Access database fields using dot syntax (requires `OverloadedRecordDot`):
```haskell
db.port :: Word16 -- Port number
db.databaseName :: Text -- Database name
db.user :: Text -- Username
db.dataDirectory :: FilePath -- Data directory path
db.socketDirectory :: FilePath -- Socket directory path
```
Computed connection helpers:
```haskell
connectionSettings :: Database -> Settings -- hasql Settings
connectionString :: Database -> Text -- libpq connection string
```
### Cache Management
```haskell
clearCache :: IO () -- Clear current user's cache
clearAllCaches :: IO () -- Clear all cached data
```
## Benchmarks
Measured with [tasty-bench](https://hackage.haskell.org/package/tasty-bench) in wall-clock mode on Apple Silicon.
| Benchmark | Time | vs baseline |
|---|---|---|
| **Lifecycle** | | |
| `defaultConfig` | 796 ms | -- |
| `defaultConfig <> verboseConfig` | 812 ms | 1.02x |
| `defaultConfig <> autoExplainConfig 100` | 769 ms | 0.97x |
| **Caching** | | |
| `withConfig` (uncached) | 755 ms | -- |
| `withCached` | 434 ms | 0.58x |
| **Snapshot** | | |
| lifecycle only | 761 ms | -- |
| lifecycle + `createSnapshot` | 1.13 s | 1.48x |
| lifecycle + `createSnapshot` + `restoreSnapshot` | 1.39 s | 1.83x |
| **Connection** | | |
| hasql `acquire` + `release` | 1.64 ms | -- |
Highlights:
- Cached startup is ~42% faster than uncached (CoW copy vs full initdb)
- `verboseConfig` and `autoExplainConfig` add negligible overhead
- Snapshot create adds ~365 ms, restore adds ~265 ms
- Connection acquire/release is ~1.6 ms once the database is running
## Comparison with tmp-postgres
ephemeral-pg is a modern replacement for tmp-postgres, addressing several issues:
| Feature | tmp-postgres | ephemeral-pg |
|---------|-------------|--------------|
| Shell injection | Vulnerable | Safe (typed-process) |
| Windows support | Partial | Unix-only (explicit) |
| Socket path length | Can fail | Validated |
| GHC version | 8.0+ | 9.6+ |
| hasql integration | Via options | Native |
| initdb caching | Yes | Yes (improved) |
| Copy-on-write | Yes | Yes |
| PostgreSQL version | 9.3+ | 14+ |
## Troubleshooting
### "initdb not found"
Ensure PostgreSQL binaries are in your PATH:
```bash
# macOS (Homebrew)
export PATH="/opt/homebrew/opt/postgresql@14/bin:$PATH"
# Linux (Debian/Ubuntu)
export PATH="/usr/lib/postgresql/14/bin:$PATH"
```
### Socket path too long
Unix sockets have a path length limit (~104 bytes). ephemeral-pg uses short
paths to avoid this, but if you specify a custom socket directory, ensure
the full path is under 90 characters.
### Permission denied
Ensure you have write access to the temporary directory. By default, the
system temp directory is used.
## Tracing
OpenTelemetry tracing is available through the optional companion
package `ephemeral-pg-opentelemetry`, which lives next to this library
in the same repository. It mirrors the `EphemeralPg` lifecycle API and
emits one span per public operation:
| Wrapper | Span name |
| ---------------------------- | ------------------------------- |
| `withTraced` | `ephemeralpg.with` |
| `withCachedTraced` | `ephemeralpg.with_cached` |
| `startTraced` | `ephemeralpg.start` |
| `stopTraced` | `ephemeralpg.stop` |
| `restartTraced` | `ephemeralpg.restart` |
| `createSnapshotTraced` | `ephemeralpg.snapshot.create` |
| `restoreSnapshotTraced` | `ephemeralpg.snapshot.restore` |
| `deleteSnapshotTraced` | `ephemeralpg.snapshot.delete` |
| `dumpTraced` | `ephemeralpg.dump` |
| `restoreTraced` | `ephemeralpg.restore` |
Each span carries the standard database and server attributes
(`db.system.name`/`db.system`, `db.namespace`/`db.name`,
`server.address`/`net.peer.name`, `server.port`/`net.peer.port`) plus
library-specific ones (`ephemeralpg.port`,
`ephemeralpg.shutdown.mode`). `server.address` is the Unix socket
directory, which is what a client passes as libpq's `host`. Attribute
name selection obeys
`OTEL_SEMCONV_STABILITY_OPT_IN` exactly the way upstream database
instrumentation does — set it to `database` for stable names,
`database/dup` for both stable and legacy. Errors from
`EphemeralPg.start` are
recorded uniformly with `error.type` (constructor name), span status
`Error`, and a `recordException` event.
The wrappers are designed to nest under a parent test span. Combined
with `hs-opentelemetry-instrumentation-hspec`, a test that calls
`withTraced` produces a tree of the form:
Run tests
└─ ephemeral-pg under OpenTelemetry
└─ emits a span tree under withTraced
└─ ephemeralpg.with
├─ ephemeralpg.start
├─ ephemeralpg.with.body
└─ ephemeralpg.stop
Add the dependency to your test-suite (the package is not yet on
Hackage; reference it via the local `cabal.project`):
```cabal
build-depends:
ephemeral-pg-opentelemetry,
hs-opentelemetry-api,
hs-opentelemetry-sdk,
hs-opentelemetry-instrumentation-hspec,
```
See `ephemeral-pg-opentelemetry/test/Demo.hs` for a runnable example.
## Cleanup after a killed consumer
`start` and `startCached` sweep abandoned temporary PostgreSQL instances before
allocating a new cluster. If a consumer exits without cleanup (including SIGKILL),
its server can remain alive until the next startup or explicit sweep:
```haskell
import Data.Monoid (Last (..))
import EphemeralPg qualified as Pg
let config = Pg.defaultConfig
{ Pg.temporaryRoot = Last (Just "/tmp/my-tests") }
removedPaths <- Pg.sweepStaleInstances config
```
Create a custom temporary root before using it. The sweep returns sorted canonical
paths actually removed. Disable automatic sweeping with
`config { Pg.sweepStaleOnStart = Last (Just False) }`; explicit sweeps still run,
and new temporary instances still hold ownership locks. An absent setting enables
automatic sweeping.
Cleanup uses local filesystem locks to protect live consumers, including startup
and data replacement. It waits up to five seconds per abandoned server for fast
shutdown, without escalating to SIGKILL. Inspection and deletion add overhead, so
a large backlog can delay startup. Uninspectable or unresponsive instances remain
for a later attempt. Supported inspection platforms are Linux (`ps` and `/proc`)
and macOS (`ps` and `lsof`).
Only immediate temporary data directories are candidates. Permanent data, socket
directories, snapshots, and initialization caches are excluded. Cached startup
with permanent data uses ordinary initialization. Historical untracked clusters
need a valid PID file; live historical servers also need verified identity and
parent PID 1. Untracked directories without PID files remain untouched. The
private `.ephemeral-pg-instances-<uid>` registry retains small lock files to avoid
concurrent lock-replacement races.
## Running tests
Run all suites locally with the pinned Nix toolchain:
```bash
nix develop .#test -c cabal test all --test-show-details=direct
```
On an Apple silicon Mac, run the Linux suites with Apple containers:
```bash
container system start
./test/platform/apple-container.sh
```
The script requires `container`, `git`, `jq`, and `tar`. It uses a pinned official
Nix image and the repository's `flake.lock`, with GHC 9.12.4 and PostgreSQL from
the `test` shell. It copies tracked and non-ignored untracked files from the current
working tree into the container's Linux filesystem, then runs every Cabal test
suite as an unprivileged user. This includes real process ownership and orphan
recovery tests. No Dockerfile or Docker daemon is needed.
The dedicated `ephemeral-pg-nix-validation` container stops after each run, including
failures, and retains Nix and Cabal caches for the next run. Failed source snapshots
and Cabal logs remain inside it for inspection. The first run downloads the Linux
toolchain and builds dependencies. Defaults are four CPUs and 4 GiB of memory;
set `EPHEMERAL_PG_CPUS` and `EPHEMERAL_PG_MEMORY` when creating a container to
change those limits. Set `EPHEMERAL_PG_CONTAINER` to use a separate named container;
the script refuses to reuse a running container. Extra arguments are passed to
`cabal test`, for example `--test-options='--match "Stale instances"'`.
The `test` shell skips editor tools, Git hook installation and development-database
initialization. Linux process inspection requires `procps` on `PATH`, supplied by
this shell. Databases and build outputs stay inside the Linux container.
## License
BSD-3-Clause