packages feed

golds-gym (empty) → 0.1.0.0

raw patch · 8 files changed

+1949/−0 lines, 8 filesdep +aesondep +basedep +benchpress

Dependencies added: aeson, base, benchpress, boxes, bytestring, directory, filepath, golds-gym, hspec, hspec-core, process, statistics, text, time, vector

Files

+ LICENSE view
@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2026++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.
+ README.md view
@@ -0,0 +1,455 @@+# golds-gym 🏋️++[![CI](https://github.com/ocramz/golds-gym/actions/workflows/ci.yml/badge.svg)](https://github.com/ocramz/golds-gym/actions/workflows/ci.yml)++A Haskell golden testing framework for performance benchmarks.++## Overview++`golds-gym` allows you to define timing benchmarks that are saved to golden files the first time they run. On subsequent runs, new benchmark results are compared against the golden baselines using configurable tolerance thresholds.++**Key Features:**+- Architecture-specific golden files (different baselines per CPU/OS)+- Configurable tolerance for mean time comparison+- **Robust statistics** mode (trimmed mean, MAD, outlier detection)+- Optional variance (stddev) warnings+- Configurable warm-up iterations+- JSON-based golden files for easy inspection+- Seamless integration with hspec+++## Usage++```haskell+import Test.Hspec+import Test.Hspec.BenchGolden++main :: IO ()+main = hspec $ do+  describe "Performance" $ do+    -- Simple benchmark with defaults (100 iterations, 15% tolerance)+    benchGolden "list append" $+      return $ [1..1000] ++ [1..1000]++    -- Benchmark with custom configuration+    benchGoldenWith defaultBenchConfig+      { iterations = 500+      , tolerancePercent = 10.0+      , warmupIterations = 10+      , warnOnVarianceChange = True+      }+      "sorting" $+      return $ sort [1000, 999..1]++    -- Robust statistics mode (outlier detection, trimmed mean)+    benchGoldenWith defaultBenchConfig+      { useRobustStatistics = True+      , trimPercent = 10.0+      , outlierThreshold = 3.0+      , tolerancePercent = 10.0+      }+      "robust benchmark" $+      return $ expensiveComputation input++    -- IO benchmark+    benchGolden "file operations" $ do+      writeFile "/tmp/test" "hello"+      readFile "/tmp/test"+```++## Golden Files++Golden files are stored in `.golden/<architecture>/` with the following structure:++```+.golden/+├── aarch64-darwin-Apple_M1/+│   ├── list-append.golden+│   ├── list-append.actual+│   └── sorting.golden+└── x86_64-linux-Intel_Core_i7/+    └── list-append.golden+```++Each `.golden` file contains JSON with timing statistics:++```json+{+  "mean": 1.234,+  "stddev": 0.056,+  "median": 1.201,+  "min": 1.100,+  "max": 1.456,+  "percentiles": [[50, 1.201], [90, 1.350], [99, 1.440]],+  "architecture": "aarch64-darwin-Apple_M1",+  "timestamp": "2026-01-30T12:00:00Z",+  "trimmedMean": 1.220,+  "mad": 0.042,+  "iqr": 0.085,+  "outliers": [1.456]+}+```++## Updating Baselines++To regenerate golden files (after intentional performance changes):++```bash+GOLDS_GYM_ACCEPT=1 cabal test+# Or with stack:+GOLDS_GYM_ACCEPT=1 stack test+```++## Configuration++### BenchConfig Options++| Field | Default | Description |+|-------|---------|-------------|+| `iterations` | 100 | Number of benchmark iterations |+| `warmupIterations` | 5 | Warm-up runs (discarded) |+| `tolerancePercent` | 15.0 | Allowed mean time deviation (%) |+| `absoluteToleranceMs` | Just 0.01 | Minimum absolute tolerance in milliseconds (hybrid tolerance) |+| `warnOnVarianceChange` | True | Warn if stddev changes significantly |+| `varianceTolerancePercent` | 50.0 | Allowed stddev deviation (%) |+| `outputDir` | ".golden" | Directory for golden files |+| `failOnFirstRun` | False | Fail if no baseline exists |+| `useRobustStatistics` | False | Use robust statistics (trimmed mean, MAD) |+| `trimPercent` | 10.0 | Percentage to trim from each tail (%) |+| `outlierThreshold` | 3.0 | MAD multiplier for outlier detection |++### Hybrid Tolerance Strategy++**New in v0.2.0**: Hybrid tolerance prevents false failures from measurement noise.++The framework uses BOTH percentage and absolute tolerance by default:++```+Benchmark passes if:+  (mean_change <= ±15%) OR (abs_time_diff <= 0.01ms)+```++#### Why Hybrid Tolerance?++For extremely fast operations (< 1ms), tiny measurement noise causes huge percentage variations:++- **Baseline**: 0.001 ms+- **Actual**: 0.0015 ms  +- **Percentage difference**: +50% ❌ (fails with 15% tolerance)+- **Absolute difference**: +0.0005 ms ✅ (negligible, within 0.01ms tolerance)++The hybrid approach automatically handles this:++- **Fast operations (< 1ms)**: Absolute tolerance dominates → noise ignored+- **Slow operations (> 1ms)**: Percentage tolerance dominates → regressions caught++#### Configuration Examples++**Default (hybrid tolerance)**:+```haskell+benchGolden "fast operation" $ do+  return $ sum [1..100]+```+Passes if within ±15% **or** ±0.01ms (10 microseconds).++**Percentage-only (disable absolute tolerance)**:+```haskell+benchGoldenWith defaultBenchConfig+  { absoluteToleranceMs = Nothing+  , tolerancePercent = 20.0+  }+  "long operation" $ do+  return $ expensiveComputation input+```+Traditional percentage-only comparison.++**Strict absolute tolerance**:+```haskell+benchGoldenWith defaultBenchConfig+  { absoluteToleranceMs = Just 0.001  -- 1 microsecond+  , tolerancePercent = 10.0+  }+  "performance-critical" $ do+  return $ criticalPath data+```+Very strict for performance-critical code.++**Relaxed tolerance for noisy CI**:+```haskell+benchGoldenWith defaultBenchConfig+  { absoluteToleranceMs = Just 0.1  -- 100 microseconds+  , tolerancePercent = 25.0+  }+  "ci benchmark" $ do+  return $ computation input+```+More forgiving for shared CI runners.++## Architecture Detection++The framework automatically detects:+- CPU architecture (x86_64, aarch64)+- Operating system (darwin, linux, windows)+- CPU model (Apple M1, Intel Core i7, etc.)++This ensures benchmarks are only compared against baselines from equivalent hardware.++## Robust Statistics++**New in 0.1.0**: Robust statistical methods for more reliable benchmark comparisons.++### Why Use Robust Statistics?++Standard mean and standard deviation are sensitive to outliers. A single anomalous timing (e.g., from GC, OS scheduling) can skew results. Robust statistics provide:++- **Trimmed Mean**: Removes extreme values before averaging+- **MAD (Median Absolute Deviation)**: Outlier-resistant measure of variance+- **Outlier Detection**: Identifies and reports anomalous timings+- **IQR (Interquartile Range)**: Spread of the middle 50% of data++### Enabling Robust Mode++```haskell+benchGoldenWith defaultBenchConfig+  { useRobustStatistics = True  -- Enable robust statistics+  , trimPercent = 10.0          -- Trim 10% from each tail+  , outlierThreshold = 3.0      -- Outliers are 3+ MADs from median+  , tolerancePercent = 10.0     -- Compare trimmed means+  }+  "my benchmark" $ do+  -- your code here+```++### How It Works++1. **Trimmed Mean**: Sorts all timing measurements, removes the top and bottom `trimPercent`, then computes the mean of remaining values.++2. **MAD Calculation**: Computes `median(|x - median(x)|)` - more robust than standard deviation.++3. **Outlier Detection**: Any measurement where `|x - median| > outlierThreshold * MAD` is flagged as an outlier.++4. **Comparison**: When enabled, uses trimmed mean instead of mean for regression detection, and MAD instead of stddev for variance checks.++### Outlier Warnings++When outliers are detected, you'll see warnings in test output:++```+Warnings:+  ⚠ 3 outlier(s) detected: 2.1ms 2.3ms 2.5ms+```++Outliers are reported but **not removed** - they're preserved in golden files for analysis.++### When to Use Robust Statistics++✅ **Use robust statistics when:**+- Benchmarking in noisy environments (shared CI runners)+- Operations subject to GC pauses or OS scheduling variability+- Fast operations (< 1ms) with high relative variance (CV > 50%)+- Sorting already-sorted data or other operations with occasional slowdowns+- You see large max/stddev values with small mean times+- You need more stable baselines across runs++❌ **Standard statistics may be better when:**+- Benchmarking isolated, long-running operations+- You have dedicated benchmark hardware+- Outliers are legitimate and should be tracked++## Integration with CI++In CI environments, you may want to:++1. **Skip benchmarks** (if CI is too noisy):+   ```bash+   GOLDS_GYM_SKIP=1 cabal test+   ```++2. **Use relaxed tolerance** (for shared CI runners):+   ```haskell+   benchGoldenWith defaultBenchConfig+     { tolerancePercent = 25.0+     , absoluteToleranceMs = Just 0.1  -- 100 microseconds+     }+     "benchmark" $ ...+   ```++3. **Enable robust statistics** (outlier detection):+   ```haskell+   benchGoldenWith defaultBenchConfig+     { useRobustStatistics = True+     , tolerancePercent = 20.0+     }+     "benchmark" $ ...+   ```++## Troubleshooting++### Random Test Failures Due to Measurement Noise++**Symptom**: Tests fail intermittently with small percentage increases despite negligible absolute time differences:++```+Mean time increased by 35.5% (tolerance: 15.0%)++Metric    Actual  Baseline    Diff+------    ------  --------    ----+Mean    0.001 ms  0.000 ms  +35.5%+```++**Root Cause**: Operations taking < 1ms have high relative measurement noise. A 0.0005ms difference is negligible but represents 50% variation.++**Solutions**:++1. **Use hybrid tolerance (default since v0.2.0)**:+   ```haskell+   benchGolden "fast operation" $ ...+   ```+   The default `absoluteToleranceMs = Just 0.01` prevents these failures.++2. **Adjust absolute tolerance threshold**:+   ```haskell+   benchGoldenWith defaultBenchConfig+     { absoluteToleranceMs = Just 0.001  -- Stricter: 1 microsecond+     }+     "very fast operation" $ ...+   ```++3. **Increase iterations for stability**:+   ```haskell+   benchGoldenWith defaultBenchConfig+     { iterations = 500  -- More samples reduce noise+     }+     "noisy operation" $ ...+   ```++4. **Use robust statistics**:+   ```haskell+   benchGoldenWith defaultBenchConfig+     { useRobustStatistics = True  -- Outlier-resistant+     , trimPercent = 10.0+     }+     "operation with outliers" $ ...+   ```++### High Variance Warnings++**Symptom**: Warnings about variance changes despite passing benchmarks:++```+Warnings:+  ⚠ Variance increased by 65.2% (0.001 ms -> 0.002 ms, tolerance: 50.0%)+```++**Solutions**:++1. **Disable variance warnings** (if not critical):+   ```haskell+   benchGoldenWith defaultBenchConfig+     { warnOnVarianceChange = False+     }+     "benchmark" $ ...+   ```++2. **Increase variance tolerance**:+   ```haskell+   benchGoldenWith defaultBenchConfig+     { varianceTolerancePercent = 100.0  -- Allow ±100% stddev change+     }+     "benchmark" $ ...+   ```++3. **Use robust statistics** (MAD instead of stddev):+   ```haskell+   benchGoldenWith defaultBenchConfig+     { useRobustStatistics = True  -- Uses MAD, more stable+     }+     "benchmark" $ ...+   ```++### Outlier Warnings++**Symptom**: Outliers detected in benchmark runs:++```+Warnings:+  ⚠ 3 outlier(s) detected: 2.1ms 2.3ms 2.5ms+```++**Causes**:+- Garbage collection pauses+- OS scheduling interruptions+- CPU thermal throttling+- Background processes++**Solutions**:++1. **Increase outlier threshold** (less sensitive):+   ```haskell+   benchGoldenWith defaultBenchConfig+     { useRobustStatistics = True+     , outlierThreshold = 5.0  -- More forgiving (default: 3.0)+     }+     "benchmark" $ ...+   ```++2. **Increase warm-up iterations**:+   ```haskell+   benchGoldenWith defaultBenchConfig+     { warmupIterations = 20  -- Stabilize before measurement+     }+     "benchmark" $ ...+   ```++3. **Minimize system load**:+   - Close background applications+   - Disable system services during benchmarking+   - Use dedicated benchmark hardware++### Benchmarks Pass Locally But Fail in CI++**Cause**: Different architecture or noisier environment.++**Solutions**:++1. **Architecture-specific baselines**: Golden files are already per-architecture. Check that your CI architecture ID matches:+   ```bash+   GOLDS_GYM_ARCH=custom-ci-id cabal test+   ```++2. **Relaxed CI configuration**:+   ```haskell+   #ifdef CI_BUILD+   ciConfig :: BenchConfig+   ciConfig = defaultBenchConfig+     { tolerancePercent = 30.0+     , absoluteToleranceMs = Just 0.2+     , useRobustStatistics = True+     }+   #endif+   ```++3. **Skip benchmarks in CI**:+   ```yaml+   # .github/workflows/ci.yml+   - name: Run tests+     run: GOLDS_GYM_SKIP=1 stack test+   ```++### Regenerating Golden Files++**When to regenerate**:+- Intentional performance improvements/changes+- Compiler upgrades affecting code generation+- Architecture changes++**How**:+```bash+GOLDS_GYM_ACCEPT=1 stack test+```++**Warning**: Only regenerate when you've verified the performance change is expected!++## License++MIT
+ example/Spec.hs view
@@ -0,0 +1,140 @@+-- | Example benchmark golden tests demonstrating golds-gym usage.+module Main (main) where++import Control.Exception (evaluate)+import Data.List (sort)+import Test.Hspec+import Test.Hspec.BenchGolden++main :: IO ()+main = hspec spec++spec :: Spec+spec = do+  describe "List Operations" $ do+    -- Simple benchmark with default configuration+    benchGolden "list append (1000 elements)" $ do+      _ <- evaluate $ [1..1000 :: Int] ++ [1..1000]+      return ()++    -- Benchmark with more iterations for stability+    benchGoldenWith defaultBenchConfig { iterations = 200 }+      "list reverse (5000 elements)" $ do+      _ <- evaluate $ reverse [1..5000 :: Int]+      return ()++  describe "Sorting Algorithms" $ do+    -- Benchmark with tighter tolerance for critical code+    benchGoldenWith defaultBenchConfig+      { iterations = 150+      , tolerancePercent = 10.0+      , warmupIterations = 10+      }+      "sort 1000 elements" $ do+      _ <- evaluate $ sort [1000, 999..1 :: Int]+      return ()++    -- Benchmark with robust statistics for operations prone to outliers+    benchGoldenWith defaultBenchConfig+      { useRobustStatistics = True+      , trimPercent = 10.0+      , outlierThreshold = 3.0+      , warnOnVarianceChange = False+      }+      "sort already sorted" $ do+      _ <- evaluate $ sort [1..1000 :: Int]+      return ()++  describe "Numeric Operations" $ do+    benchGolden "fibonacci 30" $ do+      _ <- evaluate $ fib 30+      return ()++    benchGoldenWith defaultBenchConfig+      { iterations = 500+      , tolerancePercent = 20.0  -- Higher tolerance for fast operations+      }+      "sum of list" $ do+      _ <- evaluate $ sum [1..10000 :: Int]+      return ()++  describe "IO Operations" $ do+    benchGoldenIO "evaluate lazy thunk" $ do+      let bigList = [1..100000 :: Int]+      _ <- evaluate (length bigList)+      return ()++  describe "Robust Statistics Mode" $ do+    -- Benchmark using robust statistics (trimmed mean, MAD)+    benchGoldenWith defaultBenchConfig+      { useRobustStatistics = True+      , iterations = 100+      , trimPercent = 10.0      -- Trim 10% from each tail+      , outlierThreshold = 3.0  -- 3 MADs for outlier detection+      }+      "robust mode - list reverse" $ do+      _ <- evaluate $ reverse [1..5000 :: Int]+      return ()++    -- Benchmark with robust statistics and high outlier sensitivity+    benchGoldenWith defaultBenchConfig+      { useRobustStatistics = True+      , iterations = 200+      , trimPercent = 5.0       -- Minimal trimming+      , outlierThreshold = 2.5  -- More sensitive outlier detection+      , tolerancePercent = 10.0+      }+      "robust mode - sorting" $ do+      _ <- evaluate $ sort [1000, 999..1 :: Int]+      return ()++    -- Demonstrate outlier detection with intentionally noisy operation+    benchGoldenWith defaultBenchConfig+      { useRobustStatistics = True+      , iterations = 50+      , outlierThreshold = 2.0+      }+      "robust mode - with potential outliers" $ do+      _ <- evaluate $ fib 25+      return ()++  describe "Tolerance Configuration" $ do+    -- Hybrid tolerance (default): pass if within ±15% OR ±0.01ms+    benchGolden "hybrid tolerance - fast operation" $ do+      _ <- evaluate $ sum [1..100 :: Int]+      return ()++    -- Percentage-only tolerance: disable absolute tolerance+    -- Note: This test may fail occasionally due to measurement noise,+    -- demonstrating why hybrid tolerance is the recommended default+    benchGoldenWith defaultBenchConfig+      { absoluteToleranceMs = Nothing+      , tolerancePercent = 30.0  -- Increased to reduce false failures+      }+      "percentage-only tolerance" $ do+      _ <- evaluate $ sum [1..500 :: Int]+      return ()++    -- Strict absolute tolerance: 1 microsecond+    benchGoldenWith defaultBenchConfig+      { absoluteToleranceMs = Just 0.001  -- 1 microsecond+      , tolerancePercent = 10.0+      }+      "strict absolute tolerance" $ do+      _ <- evaluate $ fib 20+      return ()++    -- Relaxed absolute tolerance for CI environments+    benchGoldenWith defaultBenchConfig+      { absoluteToleranceMs = Just 0.1  -- 100 microseconds+      , tolerancePercent = 25.0+      }+      "relaxed tolerance for CI" $ do+      _ <- evaluate $ reverse [1..1000 :: Int]+      return ()++-- | Naive Fibonacci for benchmarking purposes.+fib :: Int -> Int+fib 0 = 0+fib 1 = 1+fib n = fib (n - 1) + fib (n - 2)
+ golds-gym.cabal view
@@ -0,0 +1,56 @@+cabal-version:      3.0+name:               golds-gym+version:            0.1.0.0+synopsis:           Golden testing framework for performance benchmarks+description:+    A Haskell framework for golden testing of timing benchmarks.+    Benchmarks are saved to golden files on first run and compared+    against on subsequent runs. Golden files are architecture-specific+    to account for hardware differences.+    .+    Based on hspec and benchpress.++license:            MIT+license-file:       LICENSE+author:             Marco Zocca+maintainer:         @ocramz+category:           Testing+build-type:         Simple+extra-doc-files:    README.md++library+    exposed-modules:+        Test.Hspec.BenchGolden+        Test.Hspec.BenchGolden.Types+        Test.Hspec.BenchGolden.Arch+        Test.Hspec.BenchGolden.Runner++    build-depends:+        base >= 4.14 && < 5,+        hspec-core >= 2.10 && < 3,+        benchpress >= 0.2 && < 0.3,+        aeson >= 2.0 && < 3,+        directory >= 1.3 && < 2,+        filepath >= 1.4 && < 2,+        time >= 1.9 && < 2,+        process >= 1.6 && < 2,+        text >= 1.2 && < 3,+        bytestring >= 0.10 && < 0.13,+        statistics >= 0.16 && < 0.17,+        vector >= 0.12 && < 0.14,+        boxes >= 0.1 && < 0.2++    hs-source-dirs:   src+    default-language: Haskell2010+    ghc-options:      -Wall++test-suite golds-gym-example+    type:             exitcode-stdio-1.0+    main-is:          Spec.hs+    hs-source-dirs:   example+    default-language: Haskell2010+    ghc-options:      -Wall+    build-depends:+        base >= 4.14 && < 5,+        golds-gym,+        hspec >= 2.10 && < 3
+ src/Test/Hspec/BenchGolden.hs view
@@ -0,0 +1,422 @@+{-# LANGUAGE FlexibleInstances #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RecordWildCards #-}+{-# LANGUAGE TypeFamilies #-}+{-# OPTIONS_GHC -Wno-orphans #-}++-- |+-- Module      : Test.Hspec.BenchGolden+-- Description : Golden testing for performance benchmarks+-- Copyright   : (c) 2026+-- License     : MIT+-- Maintainer  : your.email@example.com+--+-- = Overview+--+-- @golds-gym@ is a framework for golden testing of performance benchmarks.+-- It integrates with hspec and uses benchpress for lightweight timing measurements.+--+-- Optionally, benchmarks can use robust statistics to mitigate the impact of outliers.+--+-- = Quick Start+--+-- @+-- import Test.Hspec+-- import Test.Hspec.BenchGolden+--+-- main :: IO ()+-- main = hspec $ do+--   describe \"Performance\" $ do+--     `benchGolden` "my algorithm" $+--       return $ myAlgorithm input+-- @+--+-- = How It Works+--+-- 1. On first run, the benchmark is executed and results are saved to a+--    golden file as the baseline.+--+-- 2. On subsequent runs, the benchmark is executed and compared against+--    the baseline using a configurable tolerance (default ±15%).+--+-- 3. If the mean time exceeds the tolerance, the test fails with a+--    regression or improvement notification.+--+-- = Architecture-Specific Baselines+--+-- Golden files are stored per-architecture to ensure benchmarks are only+-- compared against equivalent hardware. The architecture identifier includes+-- CPU type, OS, and CPU model.+--+-- = Configuration+--+-- Use 'benchGoldenWith' with a custom 'BenchConfig' to adjust:+--+-- * Number of iterations+-- * Warm-up iterations+-- * Tolerance percentage+-- * Absolute tolerance (hybrid tolerance strategy)+-- * Variance warnings+-- * Robust statistics mode (trimmed mean, MAD, outlier detection)+--+-- == Tolerance Configuration+--+-- The framework supports two tolerance mechanisms that work together:+--+-- 1. __Percentage tolerance__ ('tolerancePercent'): Checks if the mean time+--    change is within ±X% of the baseline. This is the traditional approach+--    and works well for operations that take more than a few milliseconds.+--+-- 2. __Absolute tolerance__ ('absoluteToleranceMs'): Checks if the absolute+--    time difference is within X milliseconds. This prevents false failures+--    for extremely fast operations (< 1ms) where measurement noise causes+--    large percentage variations despite negligible absolute differences.+--+-- By default, benchmarks pass if __EITHER__ tolerance is satisfied:+--+-- @+-- pass = (percentChange <= 15%) OR (absTimeDiff <= 0.01 ms)+-- @+--+-- This hybrid strategy combines the benefits of both approaches:+--+-- * For fast operations (< 1ms): Absolute tolerance dominates, preventing noise+-- * For slow operations (> 1ms): Percentage tolerance dominates, catching real regressions+--+-- To disable absolute tolerance and use percentage-only comparison:+--+-- @+-- benchGoldenWith defaultBenchConfig+--   { absoluteToleranceMs = Nothing+--   }+--   \"benchmark\" $ ...+-- @+--+-- To adjust the absolute tolerance threshold:+--+-- @+-- benchGoldenWith defaultBenchConfig+--   { absoluteToleranceMs = Just 0.001  -- 1 microsecond (very strict)+--   }+--   \"benchmark\" $ ...+-- @+--+-- = Environment Variables+--+-- * @GOLDS_GYM_ACCEPT=1@ - Regenerate all golden files+-- * @GOLDS_GYM_SKIP=1@ - Skip all benchmark tests+-- * @GOLDS_GYM_ARCH=custom-id@ - Override architecture detection++module Test.Hspec.BenchGolden+  ( -- * Spec Combinators+    benchGolden+  , benchGoldenWith+  , benchGoldenIO+  , benchGoldenIOWith++    -- * Configuration+  , BenchConfig(..)+  , defaultBenchConfig++    -- * Types+  , BenchGolden(..)+  , GoldenStats(..)+  , BenchResult(..)+  , Warning(..)+  , ArchConfig(..)++    -- * Low-Level API+  , runBenchGolden++    -- * Re-exports+  , module Test.Hspec.BenchGolden.Arch+  ) where++import Data.IORef+import qualified Data.Text as T+import System.Environment (lookupEnv)+import Text.Printf (printf)+import qualified Text.PrettyPrint.Boxes as Box++import Test.Hspec.Core.Spec++import Test.Hspec.BenchGolden.Arch+import Test.Hspec.BenchGolden.Runner (runBenchGolden, setAcceptGoldens, setSkipBenchmarks)+import Test.Hspec.BenchGolden.Types++-- | Create a benchmark golden test with default configuration.+--+-- This is the simplest way to add a benchmark test:+--+-- @+-- describe "Sorting" $ do+--   benchGolden "quicksort 1000 elements" $+--     return $ quicksort [1000, 999..1]+-- @+--+-- Default configuration:+--+-- * 100 iterations+-- * 5 warm-up iterations+-- * 15% tolerance+-- * Variance warnings enabled+-- * Standard statistics (not robust mode)+benchGolden :: +    String  -- ^ Name of the benchmark+    -> IO () -- ^ The IO action to benchmark+    -> Spec+benchGolden name action = benchGoldenWith defaultBenchConfig name action++-- | Create a benchmark golden test with custom configuration.+--+-- Examples:+--+-- @+-- -- Tighter tolerance for critical code+-- benchGoldenWith defaultBenchConfig+--   { iterations = 500+--   , tolerancePercent = 5.0+--   , warmupIterations = 20+--   }+--   "hot loop" $+--   return $ criticalFunction input+--+-- -- Robust statistics mode for noisy environments+-- benchGoldenWith defaultBenchConfig+--   { useRobustStatistics = True+--   , trimPercent = 10.0+--   , outlierThreshold = 3.0+--   }+--   "benchmark with outliers" $+--   return $ computation input+-- @+benchGoldenWith :: BenchConfig  -- ^ Configuration parameters+    -> String -- ^ Name of the benchmark+    -> IO () -- ^ The IO action to benchmark+    -> Spec+benchGoldenWith config name action =+  it name $ BenchGolden+    { benchName   = name+    , benchAction = action+    , benchConfig = config+    }++-- | Create a benchmark golden test for an IO action.+--+-- This is an alias for 'benchGolden' that makes it clear the action+-- involves IO (e.g., file operations, network calls).+--+-- @+-- benchGoldenIO "file read" $ do+--   contents <- readFile "large-file.txt"+--   evaluate (length contents)+-- @+--+-- Note: For IO actions in noisy environments (CI, shared systems),+-- consider using 'benchGoldenIOWith' with @useRobustStatistics = True@.+benchGoldenIO :: String -- ^ Name of the benchmark+    -> IO () -- ^ The IO action to benchmark+    -> Spec+benchGoldenIO = benchGolden++-- | Create an IO benchmark golden test with custom configuration.+benchGoldenIOWith :: BenchConfig -- ^ Configuration parameters+    -> String -- ^ Name of the benchmark+    -> IO () -- ^ The IO action to benchmark+    -> Spec+benchGoldenIOWith = benchGoldenWith++-- | Instance for BenchGolden without arguments.+instance Example BenchGolden where+  type Arg BenchGolden = ()+  evaluateExample bg params hook progress =+    evaluateExample (\() -> bg) params hook progress++-- | Instance for BenchGolden with an argument.+--+-- This allows benchmarks to receive setup data from @before@ or @around@ combinators.+instance Example (arg -> BenchGolden) where+  type Arg (arg -> BenchGolden) = arg+  evaluateExample bgFn _params hook _progress = do+    -- Read environment variables to determine accept/skip flags+    acceptEnv <- lookupEnv "GOLDS_GYM_ACCEPT"+    skipEnv <- lookupEnv "GOLDS_GYM_SKIP"+    +    let shouldAccept = case acceptEnv of+          Just "1"    -> True+          Just "true" -> True+          Just "yes"  -> True+          _           -> False+        shouldSkip = case skipEnv of+          Just "1"    -> True+          Just "true" -> True+          Just "yes"  -> True+          _           -> False+    +    -- Store the flags so Runner can access them+    setAcceptGoldens shouldAccept+    setSkipBenchmarks shouldSkip+    +    ref <- newIORef (Result "" Success)+    hook $ \arg -> do+      let bg = bgFn arg+      result <- runBenchGolden bg+      writeIORef ref (fromBenchResult result)+    readIORef ref++-- | Convert a benchmark result to an hspec Result.+fromBenchResult :: BenchResult -> Result+fromBenchResult result = case result of+  FirstRun stats ->+    Result (formatFirstRun stats) Success++  Pass golden actual warnings ->+    let info = formatPass golden actual+        warningInfo = formatWarnings warnings+    in Result (info ++ warningInfo) Success++  Regression golden actual pctChange tolerance absToleranceMs ->+    let toleranceDesc :: String+        toleranceDesc = case absToleranceMs of+          Nothing -> printf "tolerance: %.1f%%" tolerance+          Just absMs -> printf "tolerance: %.1f%% or %.3f ms" tolerance absMs+        message = printf "Mean time increased by %.1f%% (%s)\n\n%s"+                    pctChange toleranceDesc (formatRegression golden actual)+    in Result message (Failure Nothing (Reason message))++  Improvement golden actual pctChange tolerance absToleranceMs ->+    let toleranceDesc :: String+        toleranceDesc = case absToleranceMs of+          Nothing -> printf "tolerance: %.1f%%" tolerance+          Just absMs -> printf "tolerance: %.1f%% or %.3f ms" tolerance absMs+    in Result (printf "Performance improved by %.1f%% (%s)\n%s"+                pctChange toleranceDesc (formatPass golden actual))+      Success++-- | Format statistics for the first run.+formatFirstRun :: GoldenStats -> String+formatFirstRun stats = "First run - baseline created\n" ++ formatStats stats++-- | Format a regression comparison with full details.+formatRegression :: GoldenStats -> GoldenStats -> String+formatRegression golden actual =+  let meanDiff = if statsMean golden == 0+                 then 0+                 else ((statsMean actual - statsMean golden) / statsMean golden) * 100+      stddevDiff = if statsStddev golden == 0+                   then 0+                   else ((statsStddev actual - statsStddev golden) / statsStddev golden) * 100+      medianDiff = if statsMedian golden == 0+                   then 0+                   else ((statsMedian actual - statsMedian golden) / statsMedian golden) * 100+      +      -- Create detailed comparison table+      metricCol = Box.vcat Box.left $ map Box.text +        ["Metric", "------", "Mean", "Stddev", "Median", "Min", "Max"]+      actualCol = Box.vcat Box.right $ map Box.text +        [ "Actual"+        , "------"+        , printf "%.3f ms" (statsMean actual)+        , printf "%.3f ms" (statsStddev actual)+        , printf "%.3f ms" (statsMedian actual)+        , printf "%.3f ms" (statsMin actual)+        , printf "%.3f ms" (statsMax actual)+        ]+      baselineCol = Box.vcat Box.right $ map Box.text+        [ "Baseline"+        , "--------"+        , printf "%.3f ms" (statsMean golden)+        , printf "%.3f ms" (statsStddev golden)+        , printf "%.3f ms" (statsMedian golden)+        , printf "%.3f ms" (statsMin golden)+        , printf "%.3f ms" (statsMax golden)+        ]+      diffCol = Box.vcat Box.right $ map Box.text+        [ "Diff"+        , "----"+        , printf "%+.1f%%" meanDiff+        , printf "%+.1f%%" stddevDiff+        , printf "%+.1f%%" medianDiff+        , ""+        , ""+        ]+      +      table = Box.hsep 2 Box.top [metricCol, actualCol, baselineCol, diffCol]+  in Box.render table++-- | Format a passing comparison.+formatPass :: GoldenStats -> GoldenStats -> String+formatPass golden actual =+  let meanDiff = if statsMean golden == 0+                 then 0+                 else ((statsMean actual - statsMean golden) / statsMean golden) * 100+      stddevDiff = if statsStddev golden == 0+                   then 0+                   else ((statsStddev actual - statsStddev golden) / statsStddev golden) * 100+      +      -- Create table with metric, actual, baseline, and diff columns+      metricCol = Box.vcat Box.left $ map Box.text ["Metric", "------", "Mean", "Stddev"]+      actualCol = Box.vcat Box.right $ map Box.text +        [ "Actual"+        , "------"+        , printf "%.3f ms" (statsMean actual)+        , printf "%.3f ms" (statsStddev actual)+        ]+      baselineCol = Box.vcat Box.right $ map Box.text+        [ "Baseline"+        , "--------"+        , printf "%.3f ms" (statsMean golden)+        , printf "%.3f ms" (statsStddev golden)+        ]+      diffCol = Box.vcat Box.right $ map Box.text+        [ "Diff"+        , "----"+        , printf "%+.1f%%" meanDiff+        , printf "%+.1f%%" stddevDiff+        ]+      +      table = Box.hsep 2 Box.top [metricCol, actualCol, baselineCol, diffCol]+  in Box.render table++-- | Format statistics for display.+formatStats :: GoldenStats -> String+formatStats GoldenStats{..} =+  let metricCol = Box.vcat Box.left $ map Box.text+        [ "Metric", "------", "Mean", "Stddev", "Median", "Min", "Max", "Arch" ]+      valueCol = Box.vcat Box.right $ map Box.text+        [ "Value"+        , "-----"+        , printf "%.3f ms" statsMean+        , printf "%.3f ms" statsStddev+        , printf "%.3f ms" statsMedian+        , printf "%.3f ms" statsMin+        , printf "%.3f ms" statsMax+        , T.unpack statsArch+        ]+      table = Box.hsep 2 Box.top [metricCol, valueCol]+  in Box.render table++-- | Format warnings for display.+formatWarnings :: [Warning] -> String+formatWarnings [] = ""+formatWarnings ws = "\nWarnings:\n" ++ unlines (map formatWarning ws)++-- | Format a single warning.+formatWarning :: Warning -> String+formatWarning w = case w of+  VarianceIncreased golden actual pct tolerance ->+    printf "  ⚠ Variance increased by %.1f%% (%.3f ms -> %.3f ms, tolerance: %.1f%%)"+      pct golden actual tolerance++  VarianceDecreased golden actual pct tolerance ->+    printf "  ⚠ Variance decreased by %.1f%% (%.3f ms -> %.3f ms, tolerance: %.1f%%)"+      pct golden actual tolerance++  HighVariance cv ->+    printf "  ⚠ High variance detected (CV = %.1f%%)" (cv * 100)++  OutliersDetected count outliers ->+    let outlierStr = if count <= 5+                     then unwords (map (printf "%.3fms") outliers)+                     else unwords (map (printf "%.3fms") (take 5 outliers)) ++ "..."+    in printf "  ⚠ %d outlier(s) detected: %s" count outlierStr
+ src/Test/Hspec/BenchGolden/Arch.hs view
@@ -0,0 +1,182 @@+{-# LANGUAGE CPP #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}++-- |+-- Module      : Test.Hspec.BenchGolden.Arch+-- Description : Architecture detection for machine-specific golden files+-- Copyright   : (c) 2026+-- License     : MIT+-- Maintainer  : your.email@example.com+--+-- This module provides functions for detecting the current machine's+-- architecture, which is used to create architecture-specific golden files.+--+-- The architecture identifier includes:+--+-- * CPU architecture (x86_64, aarch64, etc.)+-- * Operating system (darwin, linux, windows)+-- * CPU model when available (Apple M1, Intel Core i7, etc.)++module Test.Hspec.BenchGolden.Arch+  ( -- * Architecture Detection+    detectArchitecture+  , getArchId++    -- * Environment Overrides+  , getArchFromEnv++    -- * Utilities+  , sanitizeForFilename+  ) where++import Control.Exception (catch, SomeException)+import Data.Char (isAlphaNum)+import Data.Text (Text)+import qualified Data.Text as T+import System.Environment (lookupEnv)+import System.Info (arch, os)+import System.Process (readProcess)++import Test.Hspec.BenchGolden.Types (ArchConfig(..))++-- | Detect the current machine's architecture.+--+-- This function queries the system for CPU architecture, OS, and CPU model.+-- The resulting 'ArchConfig' can be used to generate architecture-specific+-- golden file paths.+--+-- The architecture can be overridden by setting the @GOLDS_GYM_ARCH@+-- environment variable.+detectArchitecture :: IO ArchConfig+detectArchitecture = do+  envArch <- getArchFromEnv+  case envArch of+    Just customArch -> return $ ArchConfig+      { archId    = customArch+      , archOS    = T.pack os+      , archCPU   = T.pack arch+      , archModel = Just customArch+      }+    Nothing -> do+      model <- getCPUModel+      let archConfig = ArchConfig+            { archId    = buildArchId (T.pack arch) (T.pack os) model+            , archOS    = T.pack os+            , archCPU   = T.pack arch+            , archModel = model+            }+      return archConfig++-- | Build an architecture identifier from components.+buildArchId :: Text -> Text -> Maybe Text -> Text+buildArchId cpu osName maybeModel =+  let base = cpu <> "-" <> osName+  in case maybeModel of+       Nothing    -> base+       Just model -> base <> "-" <> sanitizeForFilename model++-- | Get the architecture identifier string.+--+-- This is a convenience function that returns just the ID string+-- suitable for use in file paths.+getArchId :: IO Text+getArchId = archId <$> detectArchitecture++-- | Check for architecture override from environment.+--+-- Users can set @GOLDS_GYM_ARCH@ to force a specific architecture+-- identifier, useful for CI environments with consistent hardware.+getArchFromEnv :: IO (Maybe Text)+getArchFromEnv = fmap T.pack <$> lookupEnv "GOLDS_GYM_ARCH"++-- | Get the CPU model name.+--+-- This is platform-specific:+--+-- * macOS: Uses @sysctl -n machdep.cpu.brand_string@+-- * Linux: Parses @\/proc\/cpuinfo@+-- * Windows: Uses @wmic cpu get name@+-- * Other: Returns 'Nothing'+getCPUModel :: IO (Maybe Text)+getCPUModel = do+#if defined(darwin_HOST_OS)+  getDarwinCPUModel+#elif defined(linux_HOST_OS)+  getLinuxCPUModel+#elif defined(mingw32_HOST_OS)+  getWindowsCPUModel+#else+  return Nothing+#endif++#if defined(darwin_HOST_OS)+-- | Get CPU model on macOS using sysctl.+getDarwinCPUModel :: IO (Maybe Text)+getDarwinCPUModel = do+  result <- safeReadProcess "sysctl" ["-n", "machdep.cpu.brand_string"] ""+  case result of+    Nothing -> do+      -- Apple Silicon doesn't have brand_string, try chip info+      chipResult <- safeReadProcess "sysctl" ["-n", "machdep.cpu.brand"] ""+      case chipResult of+        Nothing -> do+          -- Last resort: check if it's Apple Silicon+          armResult <- safeReadProcess "uname" ["-m"] ""+          case armResult of+            Just m | "arm" `T.isInfixOf` T.toLower m -> return $ Just "Apple_Silicon"+            _ -> return Nothing+        Just chip -> return $ Just $ cleanCPUName chip+    Just name -> return $ Just $ cleanCPUName name+#endif++#if defined(linux_HOST_OS)+-- | Get CPU model on Linux by parsing /proc/cpuinfo.+getLinuxCPUModel :: IO (Maybe Text)+getLinuxCPUModel = do+  result <- safeReadProcess "grep" ["-m1", "model name", "/proc/cpuinfo"] ""+  case result of+    Nothing -> return Nothing+    Just line ->+      let parts = T.splitOn ":" line+      in case parts of+           [_, name] -> return $ Just $ cleanCPUName name+           _         -> return Nothing+#endif++#if defined(mingw32_HOST_OS)+-- | Get CPU model on Windows using WMIC.+getWindowsCPUModel :: IO (Maybe Text)+getWindowsCPUModel = do+  result <- safeReadProcess "wmic" ["cpu", "get", "name"] ""+  case result of+    Nothing -> return Nothing+    Just output ->+      let ls = filter (not . T.null) $ T.lines output+      in case drop 1 ls of  -- Skip header line+           (name:_) -> return $ Just $ cleanCPUName name+           _        -> return Nothing+#endif++-- | Safely run a process, returning Nothing on failure.+safeReadProcess :: FilePath -> [String] -> String -> IO (Maybe Text)+safeReadProcess cmd args input =+  (Just . T.pack <$> readProcess cmd args input)+    `catch` (\(_ :: SomeException) -> return Nothing)++-- | Clean up a CPU name for use as an identifier.+cleanCPUName :: Text -> Text+cleanCPUName = T.strip . T.unwords . T.words++-- | Sanitize a string for use in filenames.+--+-- Replaces spaces with underscores and removes problematic characters.+sanitizeForFilename :: Text -> Text+sanitizeForFilename = T.map sanitizeChar+  where+    sanitizeChar c+      | isAlphaNum c = c+      | c == '-'     = c+      | c == '_'     = c+      | c == ' '     = '_'+      | otherwise    = '_'
+ src/Test/Hspec/BenchGolden/Runner.hs view
@@ -0,0 +1,440 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RecordWildCards #-}++-- |+-- Module      : Test.Hspec.BenchGolden.Runner+-- Description : Benchmark execution and golden file comparison+-- Copyright   : (c) 2026+-- License     : MIT+-- Maintainer  : your.email@example.com+--+-- This module handles running benchmarks and comparing results against+-- golden files. It includes:+--+-- * Benchmark execution with warm-up iterations+-- * Golden file I/O (reading/writing JSON statistics)+-- * Tolerance-based comparison with variance warnings+-- * Support for updating baselines via GOLDS_GYM_ACCEPT environment variable++module Test.Hspec.BenchGolden.Runner+  ( -- * Running Benchmarks+    runBenchGolden+  , runBenchmark+  , runBenchmarkWithRawTimings++    -- * Golden File Operations+  , readGoldenFile+  , writeGoldenFile+  , writeActualFile+  , getGoldenPath+  , getActualPath++    -- * Comparison+  , compareStats+  , checkVariance++    -- * Robust Statistics+  , calculateRobustStats+  , calculateTrimmedMean+  , calculateMAD+  , calculateIQR+  , detectOutliers++    -- * Environment+  , shouldUpdateGolden+  , shouldSkipBenchmarks+  , setAcceptGoldens+  , setSkipBenchmarks+  ) where++import Control.Monad (when, replicateM_)+import Data.Aeson (eitherDecodeFileStrict, encodeFile)+import Data.IORef (IORef, newIORef, readIORef, writeIORef)+import Data.List (sort)+import qualified Data.Text as T+import Data.Time (getCurrentTime)+import qualified Data.Vector.Unboxed as V+import qualified Statistics.Sample as Stats+import System.CPUTime (getCPUTime)+import System.Directory (createDirectoryIfMissing, doesFileExist)+import System.FilePath ((</>), (<.>))+import System.IO.Unsafe (unsafePerformIO)++import qualified Test.BenchPress as BP++import Test.Hspec.BenchGolden.Arch (detectArchitecture, sanitizeForFilename)+import Test.Hspec.BenchGolden.Types++-- | Run a benchmark golden test.+--+-- This function:+--+-- 1. Runs warm-up iterations (discarded)+-- 2. Runs the actual benchmark+-- 3. Writes actual results to @.actual@ file+-- 4. If no golden exists, creates it (first run)+-- 5. Otherwise, compares against golden with tolerance+--+-- The result includes any warnings (e.g., variance changes).+runBenchGolden :: BenchGolden -> IO BenchResult+runBenchGolden BenchGolden{..} = do+  -- Check for skip/update flags+  skip <- shouldSkipBenchmarks+  if skip+    then do+      -- Return a pass with dummy stats when skipped+      now <- getCurrentTime+      arch <- detectArchitecture+      let dummyStats = GoldenStats 0 0 0 0 0 [] (archId arch) now 0 0 0 []+      return $ Pass dummyStats dummyStats []+    else do+      -- Detect architecture for path construction+      arch <- detectArchitecture+      let archDir = T.unpack $ archId arch+          config  = benchConfig++      -- Create output directory+      let dir = outputDir config </> archDir+      createDirectoryIfMissing True dir++      -- Run warm-up iterations+      when (warmupIterations config > 0) $+        replicateM_ (warmupIterations config) benchAction++      -- Run the actual benchmark+      actualStats <- runBenchmark benchName benchAction config arch++      -- Write actual results+      writeActualFile (outputDir config) archDir benchName actualStats++      -- Check if we should force update+      update <- shouldUpdateGolden++      -- Read or create golden file+      let goldenPath = getGoldenPath (outputDir config) archDir benchName+      goldenExists <- doesFileExist goldenPath++      if update || not goldenExists+        then do+          -- First run or forced update: create/update golden+          writeGoldenFile (outputDir config) archDir benchName actualStats+          return $ FirstRun actualStats+        else do+          -- Compare against existing golden+          goldenResult <- readGoldenFile goldenPath+          case goldenResult of+            Left err -> error $ "Failed to read golden file: " ++ err+            Right goldenStats -> do+              let result = compareStats config goldenStats actualStats+              return result++-- | Run a benchmark and collect statistics.+runBenchmark :: String -> IO () -> BenchConfig -> ArchConfig -> IO GoldenStats+runBenchmark _name action config arch = do+  if useRobustStatistics config+    then runBenchmarkWithRawTimings _name action config arch+    else do+      -- Use benchpress for standard statistics+      (cpuStats, _wallStats) <- BP.benchmark+        (iterations config)+        (pure ())                    -- setup+        (const action)               -- action+        (const $ pure ())            -- teardown++      now <- getCurrentTime++      return GoldenStats+        { statsMean        = BP.mean cpuStats+        , statsStddev      = BP.stddev cpuStats+        , statsMedian      = BP.median cpuStats+        , statsMin         = BP.min cpuStats+        , statsMax         = BP.max cpuStats+        , statsPercentiles = BP.percentiles cpuStats+        , statsArch        = archId arch+        , statsTimestamp   = now+        , statsTrimmedMean = 0.0  -- Not calculated in non-robust mode+        , statsMAD         = 0.0+        , statsIQR         = 0.0+        , statsOutliers    = []+        }++-- | Run a benchmark with raw timing collection for robust statistics.+runBenchmarkWithRawTimings :: String -> IO () -> BenchConfig -> ArchConfig -> IO GoldenStats+runBenchmarkWithRawTimings _name action config arch = do+  -- Collect raw CPU timings+  timings <- mapM (const measureCPUTime) [1 .. iterations config]+  +  let sortedTimings = sort timings+      vec = V.fromList sortedTimings+      +      -- Standard statistics+      mean' = Stats.mean vec+      stddev' = Stats.stdDev vec+      -- Median: middle element of sorted vector+      median' = if V.null vec+                then 0.0+                else vec V.! (V.length vec `div` 2)+      min' = V.minimum vec+      max' = V.maximum vec+      +      -- Percentiles (matching benchpress format)+      percentiles' = [(p, quantile p vec) | p <- [50, 66, 75, 80, 90, 95, 98, 99, 100]]+      +      -- Robust statistics+      (trimmedMean', mad', iqr', outliers') = calculateRobustStats config vec median'++  now <- getCurrentTime++  return GoldenStats+    { statsMean        = mean'+    , statsStddev      = stddev'+    , statsMedian      = median'+    , statsMin         = min'+    , statsMax         = max'+    , statsPercentiles = percentiles'+    , statsArch        = archId arch+    , statsTimestamp   = now+    , statsTrimmedMean = trimmedMean'+    , statsMAD         = mad'+    , statsIQR         = iqr'+    , statsOutliers    = outliers'+    }+  where+    measureCPUTime = do+      startCpu <- getCPUTime+      action+      endCpu <- getCPUTime+      let cpuTime = picosToMillis (endCpu - startCpu)+      return cpuTime+    +    picosToMillis :: Integer -> Double+    picosToMillis t = realToFrac t / (10^(9 :: Int))+    +    quantile :: Int -> V.Vector Double -> Double+    quantile p v =+      let idx = ceiling ((fromIntegral (V.length v) / 100) * fromIntegral p :: Double) - 1+          safeIdx = max 0 (min (V.length v - 1) idx)+      in V.unsafeIndex v safeIdx++-- | Calculate robust statistics from raw timing data.+--+-- Returns: (trimmed mean, MAD, IQR, outliers)+calculateRobustStats :: BenchConfig -> V.Vector Double -> Double -> (Double, Double, Double, [Double])+calculateRobustStats config vec median' =+  let trimmedMean' = calculateTrimmedMean (trimPercent config) vec+      mad' = calculateMAD vec median'+      iqr' = calculateIQR vec+      outliers' = detectOutliers (outlierThreshold config) vec median' mad'+  in (trimmedMean', mad', iqr', outliers')++-- | Calculate trimmed mean by removing specified percentage from each tail.+calculateTrimmedMean :: Double -> V.Vector Double -> Double+calculateTrimmedMean trimPct vec+  | V.null vec = 0.0+  | trimPct <= 0 || trimPct >= 50 = Stats.mean vec+  | otherwise =+      let n = V.length vec+          trimCount = round (fromIntegral n * trimPct / 100.0)+          trimmed = V.slice trimCount (n - 2 * trimCount) vec+      in if V.null trimmed then Stats.mean vec else Stats.mean trimmed++-- | Calculate Median Absolute Deviation (MAD).+--+-- MAD = median(|x_i - median(x)|)+calculateMAD :: V.Vector Double -> Double -> Double+calculateMAD vec med+  | V.null vec = 0.0+  | otherwise =+      let deviations = V.toList $ V.map (\x -> abs (x - med)) vec+          sortedDevs = sort deviations+          n = length sortedDevs+      in sortedDevs !! (n `div` 2)++-- | Calculate Interquartile Range (IQR = Q3 - Q1).+calculateIQR :: V.Vector Double -> Double+calculateIQR vec+  | V.length vec < 4 = 0.0+  | otherwise =+      let q1 = quantileAt 25 vec+          q3 = quantileAt 75 vec+      in q3 - q1+  where+    quantileAt :: Int -> V.Vector Double -> Double+    quantileAt p v =+      let n = V.length v+          idx = max 0 $ min (n - 1) $ round (fromIntegral n * fromIntegral p / (100.0 :: Double)) - 1+      in V.unsafeIndex v idx++-- | Detect outliers using MAD-based threshold.+--+-- An observation is an outlier if: |x - median| > threshold * MAD+detectOutliers :: Double -> V.Vector Double -> Double -> Double -> [Double]+detectOutliers threshold vec med mad+  | V.null vec = []+  | mad == 0 = []  -- No variance, no outliers+  | otherwise =+      let isOutlier x = abs (x - med) > threshold * mad+      in V.toList $ V.filter isOutlier vec++-- | Compare actual stats against golden stats.+--+-- Returns a 'BenchResult' indicating whether the benchmark passed,+-- regressed, or improved, along with any warnings.+--+-- = Hybrid Tolerance Strategy+--+-- The comparison uses BOTH percentage and absolute tolerance (when configured):+--+-- 1. Calculate percentage difference: @((actual - golden) / golden) * 100@+--+-- 2. Pass if @abs(percentDiff) <= tolerancePercent@ (percentage check)+--+-- 3. OR if @abs(actual - golden) <= absoluteToleranceMs@ (absolute check)+--+-- This prevents false failures for sub-millisecond operations where measurement+-- noise creates large percentage variations despite negligible absolute differences.+compareStats :: BenchConfig -> GoldenStats -> GoldenStats -> BenchResult+compareStats config golden actual =+  let -- Choose comparison metric based on config+      (goldenValue, actualValue) =+        if useRobustStatistics config+          then (statsTrimmedMean golden, statsTrimmedMean actual)+          else (statsMean golden, statsMean actual)++      -- Calculate percentage difference+      meanDiff = if goldenValue == 0+                 then if actualValue == 0 then 0 else 100+                 else ((actualValue - goldenValue) / goldenValue) * 100++      absDiff = abs meanDiff+      tolerance = tolerancePercent config++      -- Calculate absolute time difference (in milliseconds)+      absTimeDiff = abs (actualValue - goldenValue)++      -- Check if within absolute tolerance (hybrid tolerance strategy)+      withinAbsoluteTolerance = case absoluteToleranceMs config of+        Nothing -> False+        Just absThreshold -> absTimeDiff <= absThreshold++      -- Check variance if enabled+      baseWarnings = if warnOnVarianceChange config+                     then checkVariance config golden actual+                     else []+      +      -- Add outlier warnings if robust statistics enabled+      outlierWarnings = if useRobustStatistics config && not (null (statsOutliers actual))+                        then [OutliersDetected (length $ statsOutliers actual) (statsOutliers actual)]+                        else []+      +      warnings = baseWarnings ++ outlierWarnings++  in if absDiff <= tolerance || withinAbsoluteTolerance+     then Pass golden actual warnings+     else if meanDiff > 0+          then Regression golden actual meanDiff tolerance (absoluteToleranceMs config)+          else Improvement golden actual (abs meanDiff) tolerance (absoluteToleranceMs config)++-- | Check for variance changes and generate warnings.+checkVariance :: BenchConfig -> GoldenStats -> GoldenStats -> [Warning]+checkVariance config golden actual =+  let -- Use MAD for robust statistics, stddev otherwise+      (goldenVar, actualVar) =+        if useRobustStatistics config+          then (statsMAD golden, statsMAD actual)+          else (statsStddev golden, statsStddev actual)++      varDiff = if goldenVar == 0+                then if actualVar == 0 then 0 else 100+                else ((actualVar - goldenVar) / goldenVar) * 100++      _absVarDiff = abs varDiff+      varTolerance = varianceTolerancePercent config++      -- Coefficient of variation (CV) for high variance detection+      -- Use appropriate measure based on mode+      cv = if useRobustStatistics config+           then if statsMedian actual == 0 then 0 else statsMAD actual / statsMedian actual+           else if statsMean actual == 0 then 0 else statsStddev actual / statsMean actual++  in concat+       [ [ VarianceIncreased goldenVar actualVar varDiff varTolerance+         | varDiff > varTolerance ]+       , [ VarianceDecreased goldenVar actualVar (abs varDiff) varTolerance+         | varDiff < negate varTolerance ]+       , [ HighVariance cv | cv > 0.5 ]  -- CV > 50% is considered high+       ]++-- | Get the path for a golden file.+getGoldenPath :: FilePath -> FilePath -> String -> FilePath+getGoldenPath outDir archDir name =+  outDir </> archDir </> sanitizeName name <.> "golden"++-- | Get the path for an actual results file.+getActualPath :: FilePath -> FilePath -> String -> FilePath+getActualPath outDir archDir name =+  outDir </> archDir </> sanitizeName name <.> "actual"++-- | Sanitize a benchmark name for use in filenames.+sanitizeName :: String -> FilePath+sanitizeName = T.unpack . sanitizeForFilename . T.pack++-- | Read a golden file.+readGoldenFile :: FilePath -> IO (Either String GoldenStats)+readGoldenFile = eitherDecodeFileStrict++-- | Write a golden file.+writeGoldenFile :: FilePath -> FilePath -> String -> GoldenStats -> IO ()+writeGoldenFile outDir archDir name stats = do+  let path = getGoldenPath outDir archDir name+  encodeFile path stats++-- | Write an actual results file.+writeActualFile :: FilePath -> FilePath -> String -> GoldenStats -> IO ()+writeActualFile outDir archDir name stats = do+  let path = getActualPath outDir archDir name+  encodeFile path stats++-- | Global state for tracking command-line flags set by hspec options.+{-# NOINLINE acceptGoldensRef #-}+acceptGoldensRef :: IORef Bool+acceptGoldensRef = unsafePerformIO $ newIORef False++{-# NOINLINE skipBenchmarksRef #-}+skipBenchmarksRef :: IORef Bool+skipBenchmarksRef = unsafePerformIO $ newIORef False++-- | Set the accept goldens flag (called from BenchGolden Example instance).+setAcceptGoldens :: Bool -> IO ()+setAcceptGoldens = writeIORef acceptGoldensRef++-- | Set the skip benchmarks flag (called from BenchGolden Example instance).+setSkipBenchmarks :: Bool -> IO ()+setSkipBenchmarks = writeIORef skipBenchmarksRef++-- | Check if golden files should be updated.+--+-- Returns 'True' if @GOLDS_GYM_ACCEPT@ environment variable is set.+-- +-- Usage:+--+-- @+-- GOLDS_GYM_ACCEPT=1 cabal test+-- GOLDS_GYM_ACCEPT=1 stack test+-- @+shouldUpdateGolden :: IO Bool+shouldUpdateGolden = readIORef acceptGoldensRef++-- | Check if benchmarks should be skipped entirely.+--+-- Returns 'True' if @GOLDS_GYM_SKIP@ environment variable is set.+-- Useful for CI environments where benchmark hardware is inconsistent.+--+-- Usage:+--+-- @+-- GOLDS_GYM_SKIP=1 cabal test+-- GOLDS_GYM_SKIP=1 stack test+-- @+shouldSkipBenchmarks :: IO Bool+shouldSkipBenchmarks = readIORef skipBenchmarksRef
+ src/Test/Hspec/BenchGolden/Types.hs view
@@ -0,0 +1,233 @@+{-# LANGUAGE DeriveGeneric #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE RecordWildCards #-}++-- |+-- Module      : Test.Hspec.BenchGolden.Types+-- Description : Core types for benchmark golden testing+-- Copyright   : (c) 2026+-- License     : MIT+-- Maintainer  : your.email@example.com+--+-- This module defines the core data types used by the golds-gym framework:+--+-- * 'BenchGolden' - Configuration for a benchmark golden test+-- * 'BenchConfig' - Configurable benchmark parameters+-- * 'GoldenStats' - Statistics stored in golden files+-- * 'ArchConfig' - Machine architecture identification+-- * 'BenchResult' - Result of comparing benchmark against golden++module Test.Hspec.BenchGolden.Types+  ( -- * Benchmark Configuration+    BenchGolden(..)+  , BenchConfig(..)+  , defaultBenchConfig++    -- * Golden File Statistics+  , GoldenStats(..)++    -- * Architecture Configuration+  , ArchConfig(..)++    -- * Benchmark Results+  , BenchResult(..)+  , Warning(..)+  ) where++import Data.Aeson+import Data.Text (Text)+import Data.Time (UTCTime)+import GHC.Generics (Generic)++-- | Configuration for a single benchmark golden test.+data BenchGolden = BenchGolden+  { benchName   :: !String+    -- ^ Name of the benchmark (used for golden file naming)+  , benchAction :: !(IO ())+    -- ^ The IO action to benchmark+  , benchConfig :: !BenchConfig+    -- ^ Configuration parameters+  }++-- | Configurable parameters for benchmark execution and comparison.+data BenchConfig = BenchConfig+  { iterations            :: !Int+    -- ^ Number of benchmark iterations to run+  , warmupIterations      :: !Int+    -- ^ Number of warm-up iterations (discarded before measurement)+  , tolerancePercent      :: !Double+    -- ^ Allowed deviation in mean time (as percentage, e.g., 15.0 = ±15%)+  , absoluteToleranceMs   :: !(Maybe Double)+    -- ^ Minimum absolute tolerance in milliseconds (e.g., 0.01 = 10 microseconds).+    --   When set, benchmarks pass if EITHER the percentage difference is within+    --   'tolerancePercent' OR the absolute time difference is within this threshold.+    --   This prevents false failures for extremely fast operations (< 1ms) where+    --   measurement noise causes large percentage variations despite negligible+    --   absolute time differences. Set to 'Nothing' to disable (percentage-only).+  , warnOnVarianceChange  :: !Bool+    -- ^ Whether to emit warnings when stddev changes significantly+  , varianceTolerancePercent :: !Double+    -- ^ Allowed deviation in stddev (as percentage)+  , outputDir             :: !FilePath+    -- ^ Directory for storing golden files+  , failOnFirstRun        :: !Bool+    -- ^ Whether to fail if no golden file exists yet+  , useRobustStatistics   :: !Bool+    -- ^ Use robust statistics (trimmed mean, MAD) instead of mean/stddev+  , trimPercent           :: !Double+    -- ^ Percentage to trim from each tail for trimmed mean (e.g., 10.0 = 10%)+  , outlierThreshold      :: !Double+    -- ^ MAD multiplier for outlier detection (e.g., 3.0 = 3 MADs from median)+  } deriving (Show, Eq, Generic)++-- | Default benchmark configuration with sensible defaults.+--+-- * 100 iterations+-- * 5 warm-up iterations+-- * 15% tolerance on mean time+-- * 0.01 ms (10 microseconds) absolute tolerance - prevents false failures for fast operations+-- * Variance warnings enabled at 50% tolerance+-- * Output to @.golden/@ directory+-- * Success on first run (creates baseline)+--+-- = Hybrid Tolerance Strategy+--+-- The default configuration uses BOTH percentage and absolute tolerance:+--+-- * Benchmarks pass if mean time is within ±15% OR within ±0.01ms+-- * This prevents measurement noise from failing fast operations (< 1ms)+-- * For slower operations (> 1ms), percentage tolerance dominates+--+-- Set @absoluteToleranceMs = Nothing@ for percentage-only comparison.+defaultBenchConfig :: BenchConfig+defaultBenchConfig = BenchConfig+  { iterations            = 100+  , warmupIterations      = 5+  , tolerancePercent      = 15.0+  , absoluteToleranceMs   = Just 0.01  -- 10 microseconds+  , warnOnVarianceChange  = True+  , varianceTolerancePercent = 50.0+  , outputDir             = ".golden"+  , failOnFirstRun        = False+  , useRobustStatistics   = False+  , trimPercent           = 10.0+  , outlierThreshold      = 3.0+  }++-- | Statistics stored in golden files.+--+-- These represent the baseline performance characteristics of a benchmark+-- on a specific architecture.+data GoldenStats = GoldenStats+  { statsMean        :: !Double+    -- ^ Mean execution time in milliseconds+  , statsStddev      :: !Double+    -- ^ Standard deviation in milliseconds+  , statsMedian      :: !Double+    -- ^ Median execution time in milliseconds+  , statsMin         :: !Double+    -- ^ Minimum execution time in milliseconds+  , statsMax         :: !Double+    -- ^ Maximum execution time in milliseconds+  , statsPercentiles :: ![(Int, Double)]+    -- ^ Percentile values (e.g., [(50, 1.2), (90, 1.5), (99, 1.8)])+  , statsArch        :: !Text+    -- ^ Architecture identifier+  , statsTimestamp   :: !UTCTime+    -- ^ When this baseline was recorded+  , statsTrimmedMean :: !Double+    -- ^ Trimmed mean (with tails removed) in milliseconds+  , statsMAD         :: !Double+    -- ^ Median absolute deviation in milliseconds+  , statsIQR         :: !Double+    -- ^ Interquartile range (Q3 - Q1) in milliseconds+  , statsOutliers    :: ![Double]+    -- ^ List of detected outlier timings in milliseconds+  } deriving (Show, Eq, Generic)++-- | Machine architecture configuration.+--+-- Used to generate unique identifiers for golden file directories,+-- ensuring benchmarks are only compared against equivalent hardware.+data ArchConfig = ArchConfig+  { archId    :: !Text+    -- ^ Unique identifier (e.g., "aarch64-darwin-Apple_M1")+  , archOS    :: !Text+    -- ^ Operating system (e.g., "darwin", "linux")+  , archCPU   :: !Text+    -- ^ CPU architecture (e.g., "aarch64", "x86_64")+  , archModel :: !(Maybe Text)+    -- ^ CPU model if available (e.g., "Apple M1", "Intel Core i7")+  } deriving (Show, Eq, Generic)++-- | Result of running a benchmark and comparing against golden.+data BenchResult+  = FirstRun !GoldenStats+    -- ^ No golden file existed; baseline created+  | Pass !GoldenStats !GoldenStats ![Warning]+    -- ^ Benchmark passed (golden stats, actual stats, warnings)+  | Regression !GoldenStats !GoldenStats !Double !Double !(Maybe Double)+    -- ^ Performance regression (golden, actual, percent change, tolerance, absolute tolerance)+  | Improvement !GoldenStats !GoldenStats !Double !Double !(Maybe Double)+    -- ^ Performance improvement (golden, actual, percent change, tolerance, absolute tolerance)+  deriving (Show, Eq)++-- | Warnings that may be emitted during benchmark comparison.+data Warning+  = VarianceIncreased !Double !Double !Double !Double+    -- ^ Stddev increased (golden, actual, percent change, tolerance)+  | VarianceDecreased !Double !Double !Double !Double+    -- ^ Stddev decreased significantly (golden, actual, percent change, tolerance)+  | HighVariance !Double+    -- ^ Current run has unusually high variance+  | OutliersDetected !Int ![Double]+    -- ^ Outliers detected (count, list of outlier timings)+  deriving (Show, Eq)++-- JSON instances for GoldenStats (stored in golden files)++instance ToJSON GoldenStats where+  toJSON GoldenStats{..} = object+    [ "mean"        .= statsMean+    , "stddev"      .= statsStddev+    , "median"      .= statsMedian+    , "min"         .= statsMin+    , "max"         .= statsMax+    , "percentiles" .= statsPercentiles+    , "architecture" .= statsArch+    , "timestamp"   .= statsTimestamp+    , "trimmedMean" .= statsTrimmedMean+    , "mad"         .= statsMAD+    , "iqr"         .= statsIQR+    , "outliers"    .= statsOutliers+    ]++instance FromJSON GoldenStats where+  parseJSON = withObject "GoldenStats" $ \v -> GoldenStats+    <$> v .: "mean"+    <*> v .: "stddev"+    <*> v .: "median"+    <*> v .: "min"+    <*> v .: "max"+    <*> v .: "percentiles"+    <*> v .: "architecture"+    <*> v .: "timestamp"+    <*> v .: "trimmedMean"+    <*> v .: "mad"+    <*> v .: "iqr"+    <*> v .: "outliers"++instance ToJSON ArchConfig where+  toJSON ArchConfig{..} = object+    [ "id"    .= archId+    , "os"    .= archOS+    , "cpu"   .= archCPU+    , "model" .= archModel+    ]++instance FromJSON ArchConfig where+  parseJSON = withObject "ArchConfig" $ \v -> ArchConfig+    <$> v .: "id"+    <*> v .: "os"+    <*> v .: "cpu"+    <*> v .:? "model"