golds-gym 0.3.0.0 → 0.4.0.0

raw patch · 10 files changed

+801/−161 lines, 10 filesdep +QuickCheckdep +deepseqPVP ok

version bump matches the API change (PVP)

Dependencies added: QuickCheck, deepseq

API changes (from Hackage documentation)

Files

• CHANGELOG.md +49/−0
• README.md +33/−7
• example/Spec.hs +61/−94
• golds-gym.cabal +19/−2
• src/Test/Hspec/BenchGolden.hs +127/−45
• src/Test/Hspec/BenchGolden/Arch.hs +1/−1
• src/Test/Hspec/BenchGolden/Lenses.hs +18/−1
• src/Test/Hspec/BenchGolden/Runner.hs +172/−8
• src/Test/Hspec/BenchGolden/Types.hs +9/−3
• test/Properties.hs +312/−0

@@ -1,5 +1,54 @@ # Changelog +## [0.4.0]++### Added++- **QuickCheck property tests** for statistical computations in Runner module+  - 28 property tests covering mathematical invariants and edge cases+  - Tests for `calculateTrimmedMean`, `calculateMAD`, `calculateIQR`, `detectOutliers`+  - Tests for `compareStats` tolerance logic (hybrid percentage/absolute)+  - Tests for `checkVariance` warning generation+  - Custom generators for valid timing data and benchmark configurations+  - New test suite: `golds-gym-properties`++- **Evaluation strategy combinators** for proper laziness handling+  - `nf` - Force result to normal form (deep evaluation)+  - `whnf` - Force result to weak head normal form (shallow evaluation)+  - `nfIO` - Normal form for IO actions+  - `whnfIO` - Weak head normal form for IO actions+  - `nfAppIO` - Normal form for functions returning IO+  - `whnfAppIO` - WHNF for functions returning IO+  - `io` - Plain IO action (for backward compatibility)+  - Vendored evaluation loops from tasty-bench with attribution++- **`BenchAction` type** wrapping `Word64 -> IO ()` for benchmarkable actions+  - Enables direct benchmarking of pure functions without manual `evaluate` calls++### Changed++- **BREAKING:** All benchmark API functions now accept `BenchAction` instead of `IO ()`+  - `benchGolden :: String -> BenchAction -> Spec`+  - `benchGoldenWith :: BenchConfig -> String -> BenchAction -> Spec`+  - `benchGoldenWithExpectation :: String -> BenchConfig -> [Expectation] -> BenchAction -> Spec`+  - Migration: wrap existing `IO ()` actions with `io` combinator+  - New: use `nf`/`whnf` for pure functions instead of manual `evaluate`++- **Output format:** Baseline now appears before Actual in comparison tables++- **Error messages:** Tolerance values in failure messages are now extracted from expectations rather than config defaults++### Fixed++- Evaluation strategy bug where GHC could share computation across benchmark iterations+- Error message wording for performance improvements (now says "decreased by" instead of "increased by -X%")+- Flaky micro-benchmarks stabilized by increasing iteration counts (500-2000 iterations)++### Dependencies++- Added `deepseq >= 1.4 && < 2` for `NFData` constraint+- Added `QuickCheck >= 2.14 && < 3` for property tests (test suite only)+ ## [0.3.0]  ### Added

@@ -15,26 +15,49 @@ ```haskell import Test.Hspec import Test.Hspec.BenchGolden+import Data.List (sort)  main :: IO () main = hspec $ do   describe "Performance" $ do-    -- Simple benchmark (100 iterations, ±15% tolerance)+    -- Pure function with normal form evaluation (deep, full evaluation)     benchGolden "list append" $-      return $ [1..1000] ++ [1..1000]+      nf (\xs -> xs ++ xs) [1..1000] +    -- Weak head normal form (shallow, outermost constructor only)+    benchGolden "replicate" $+      whnf (replicate 1000) 42+     -- Custom configuration     benchGoldenWith defaultBenchConfig       { iterations = 500       , tolerancePercent = 10.0       }       "sorting" $-      return $ sort [1000, 999..1]+      nf sort [1000, 999..1] ``` +**Evaluation strategies** (required - specify how values are forced):+- `nf f x` - Force result of `f x` to **normal form** (deep, full evaluation)+- `whnf f x` - Force result of `f x` to **weak head normal form** (shallow, outermost constructor only)+- `nfIO action` - Execute IO action and force result to normal form+- `whnfIO action` - Execute IO action and force result to WHNF+- `nfAppIO f x` - Apply function, execute resulting IO, force result to normal form+- `whnfAppIO f x` - Apply function, execute resulting IO, force result to WHNF+- `io action` - Plain IO action without additional forcing++**Why evaluation strategies matter**: Without forcing, GHC may optimize away computations or share results across iterations, making benchmarks meaningless. Use `nf` for most cases unless you specifically want lazy evaluation (`whnf`).+ **First run** creates `.golden/<arch>/list-append.golden` with baseline stats.  -**Subsequent runs** compare against baseline. Test fails if mean time changes by >15% (configurable).+**Subsequent runs** compare against baseline. Test fails if mean time changes beyond tolerance (default: ±15% OR ±0.01ms). +**Output format** :+```+Metric  Baseline    Actual      Diff+------  --------    ------      ----+Mean    0.150 ms  0.170 ms   +13.3%+```+ **Update baselines** after intentional changes: ```bash GOLDS_GYM_ACCEPT=1 stack test@@ -85,7 +108,7 @@   , outlierThreshold = 3.0      -- Flag outliers >3 MADs from median   }   "noisy benchmark" $-  return $ computation input+  nf computation input ```  **When to use:**@@ -104,14 +127,14 @@ -- Compare by median instead of mean (more robust) benchGoldenWithExpectation "median comparison" defaultBenchConfig   [expect _statsMedian (Percent 10.0)]-  myAction+  (nf myAlgorithm input)  -- Compose multiple requirements (both must pass) benchGoldenWithExpectation "strict requirements" defaultBenchConfig   [ expect _statsMean (Percent 15.0) &&~     expect _statsIQR (Absolute 0.1)     -- Low variance required   ]-  myAction+  (nf criticalFunction data) ```  **Available lenses:** `_statsMean`, `_statsMedian`, `_statsTrimmedMean`, `_statsStddev`, `_statsMAD`, `_statsIQR`, `_statsMin`, `_statsMax`@@ -130,6 +153,9 @@ - [API documentation](https://hackage.haskell.org/package/golds-gym) - Full Haddock docs - [Example benchmarks](example/Spec.hs) - Comprehensive usage examples - [CHANGELOG](CHANGELOG.md) - Version history and migration guides++## Related Work+* [tasty-bench](https://hackage.haskell.org/package/tasty-bench)  ## License 

@@ -1,11 +1,9 @@ -- | Example benchmark golden tests demonstrating golds-gym usage. module Main (main) where -import Control.Exception (evaluate) import Data.List (sort) import Test.Hspec import Test.Hspec.BenchGolden-import Test.Hspec.BenchGolden.Lenses  main :: IO () main = hspec spec@@ -13,27 +11,25 @@ spec :: Spec spec = do   describe "List Operations" $ do-    -- Simple benchmark with default configuration-    benchGolden "list append (1000 elements)" $ do-      _ <- evaluate $ [1..1000 :: Int] ++ [1..1000]-      return ()+    -- Simple benchmark with default configuration using nf for normal form+    benchGoldenWith defaultBenchConfig { iterations = 1000 }+      "list append (1000 elements)" $+      nf (\xs -> xs ++ xs) [1..1000 :: Int]      -- Benchmark with more iterations for stability-    benchGoldenWith defaultBenchConfig { iterations = 200 }-      "list reverse (5000 elements)" $ do-      _ <- evaluate $ reverse [1..5000 :: Int]-      return ()+    benchGoldenWith defaultBenchConfig { iterations = 1000 }+      "list reverse (5000 elements)" $+      nf reverse [1..5000 :: Int]    describe "Sorting Algorithms" $ do     -- Benchmark with tighter tolerance for critical code     benchGoldenWith defaultBenchConfig-      { iterations = 150+      { iterations = 1000       , tolerancePercent = 10.0       , warmupIterations = 10       }-      "sort 1000 elements" $ do-      _ <- evaluate $ sort [1000, 999..1 :: Int]-      return ()+      "sort 1000 elements" $+      nf sort [1000, 999..1 :: Int]      -- Benchmark with robust statistics for operations prone to outliers     benchGoldenWith defaultBenchConfig@@ -42,139 +38,110 @@       , outlierThreshold = 3.0       , warnOnVarianceChange = False       }-      "sort already sorted" $ do-      _ <- evaluate $ sort [1..1000 :: Int]-      return ()+      "sort already sorted" $+      nf sort [1..1000 :: Int]    describe "Numeric Operations" $ do-    benchGolden "fibonacci 30" $ do-      _ <- evaluate $ fib 30-      return ()+    benchGoldenWith defaultBenchConfig { iterations = 500 }+      "fibonacci 20" $+      nf fib 20      benchGoldenWith defaultBenchConfig-      { iterations = 500+      { iterations = 2000       , tolerancePercent = 20.0  -- Higher tolerance for fast operations       }-      "sum of list" $ do-      _ <- evaluate $ sum [1..10000 :: Int]-      return ()+      "sum of list" $+      nf (\n -> sum [1..n]) (10000 :: Int) +  describe "Best Practices: Avoiding Shared Thunks" $ do+    -- Lambda wrapper forces list reconstruction on each iteration+    -- The list [1..n] is rebuilt for every benchmark iteration+    benchGoldenWith defaultBenchConfig { iterations = 1000 }+      "proper data structure reconstruction" $+      nf (\n -> sum [1..n]) (5000 :: Int)++    benchGoldenWith defaultBenchConfig { iterations = 1000 }+      "function application with parameter" $+      nf reverse [1..5000 :: Int]++    benchGoldenWith defaultBenchConfig { iterations = 1000 }+      "nested data structure reconstruction" $+      nf (\xs -> concat (replicate 100 xs)) [1..50 :: Int]+   describe "Robust Statistics Mode" $ do     -- Benchmark using robust statistics (trimmed mean, MAD)     benchGoldenWith defaultBenchConfig       { useRobustStatistics = True-      , iterations = 100+      , iterations = 500       , trimPercent = 10.0      -- Trim 10% from each tail       , outlierThreshold = 3.0  -- 3 MADs for outlier detection       }-      "robust mode - list reverse" $ do-      _ <- evaluate $ reverse [1..5000 :: Int]-      return ()+      "robust mode - list reverse" $+      nf reverse [1..5000 :: Int]      -- Benchmark with robust statistics and high outlier sensitivity     benchGoldenWith defaultBenchConfig       { useRobustStatistics = True-      , iterations = 200+      , iterations = 1000       , trimPercent = 5.0       -- Minimal trimming       , outlierThreshold = 2.5  -- More sensitive outlier detection       , tolerancePercent = 10.0       }       "robust mode - sorting" $ do-      _ <- evaluate $ sort [1000, 999..1 :: Int]-      return ()+      nf sort [100000, 99999..1 :: Int]      -- Demonstrate outlier detection with intentionally noisy operation     benchGoldenWith defaultBenchConfig       { useRobustStatistics = True-      , iterations = 50+      , iterations = 500       , outlierThreshold = 2.0+      , tolerancePercent = 20.0       }-      "robust mode - with potential outliers" $ do-      _ <- evaluate $ fib 25-      return ()+      "robust mode - with potential outliers" $+      nf fib 25    describe "Tolerance Configuration" $ do     -- Hybrid tolerance (default): pass if within ±15% OR ±0.01ms-    benchGolden "hybrid tolerance - fast operation" $ do-      _ <- evaluate $ sum [1..100 :: Int]-      return ()+    benchGoldenWith defaultBenchConfig { iterations = 2000 }+      "hybrid tolerance - fast operation" $+      nf (\n -> sum [1..n]) (100 :: Int)      -- Percentage-only tolerance: disable absolute tolerance     -- Note: This test may fail occasionally due to measurement noise,     -- demonstrating why hybrid tolerance is the recommended default     benchGoldenWith defaultBenchConfig       { absoluteToleranceMs = Nothing-      , tolerancePercent = 30.0  -- Increased to reduce false failures+      , tolerancePercent = 50.0  -- Increased to reduce false failures+      , iterations = 2000       }-      "percentage-only tolerance" $ do-      _ <- evaluate $ sum [1..500 :: Int]-      return ()+      "percentage-only tolerance" $+      nf (\n -> sum [1..n]) (500 :: Int)      -- Strict absolute tolerance: 1 microsecond     benchGoldenWith defaultBenchConfig       { absoluteToleranceMs = Just 0.001  -- 1 microsecond       , tolerancePercent = 10.0+      , iterations = 1000       }-      "strict absolute tolerance" $ do-      _ <- evaluate $ fib 20-      return ()+      "strict absolute tolerance" $+      nf fib 20 -    -- Relaxed absolute tolerance for CI environments-    benchGoldenWith defaultBenchConfig-      { absoluteToleranceMs = Just 0.1  -- 100 microseconds-      , tolerancePercent = 25.0-      }-      "relaxed tolerance for CI" $ do-      _ <- evaluate $ reverse [1..1000 :: Int]-      return ()    describe "Lens-Based Expectations (Advanced)" $ do-    -- Median-based comparison instead of mean-    benchGoldenWithExpectation "median-based comparison" defaultBenchConfig-      [expect _statsMedian (Percent 10.0)]-      $ do-        _ <- evaluate $ sort [1000, 999..1 :: Int]-        return () -    -- Compose multiple expectations with AND-    benchGoldenWithExpectation "composed expectations (AND)" defaultBenchConfig-      [ expect _statsMean (Percent 15.0) &&~-        expect _statsMAD (Percent 50.0)-      ]-      $ do-        _ <- evaluate $ fib 25-        return ()--    -- Compose multiple expectations with OR-    benchGoldenWithExpectation "composed expectations (OR)" defaultBenchConfig-      [ expect _statsMedian (Percent 10.0) ||~-        expect _statsMin (Absolute 0.01)-      ]-      $ do-        _ <- evaluate $ sum [1..500 :: Int]-        return ()--    -- Expect reasonable performance (will pass with normal variance)-    benchGoldenWithExpectation "flexible tolerance" defaultBenchConfig-      [expect _statsMean (Percent 20.0)]  -- Wide tolerance for example-      $ do-        _ <- evaluate $ [1..1000 :: Int]  -- Trivial operation-        return ()-     -- Hybrid tolerance with custom absolute threshold-    benchGoldenWithExpectation "hybrid tolerance custom" defaultBenchConfig-      [expect _statsMean (Hybrid 20.0 0.005)]  -- ±20% OR ±5μs-      $ do-        _ <- evaluate $ sort [100, 99..1 :: Int]-        return ()+    benchGoldenWithExpectation "hybrid tolerance custom" +      defaultBenchConfig { iterations = 1000 }+      [expect _statsMean (Hybrid 5.0 0.0001)] +      $ nf (\n -> sort [n, n-1 .. 1]) (200 :: Int)      -- Use robust statistics lens (trimmed mean)     benchGoldenWithExpectation "trimmed mean comparison" -      (defaultBenchConfig { useRobustStatistics = True })-      [expect _statsTrimmedMean (Percent 20.0)]-      $ do-        _ <- evaluate $ fib 26-        return ()+      (defaultBenchConfig { useRobustStatistics = True, iterations = 1000 })+      [expect _statsTrimmedMean (Percent 25.0)]+      $ nf fib 26++  -- | Naive Fibonacci for benchmarking purposes. fib :: Int -> Int

@@ -1,6 +1,6 @@ cabal-version:      3.0 name:               golds-gym-version:            0.3.0.0+version:            0.4.0.0 synopsis:           Golden testing framework for performance benchmarks description:     A Haskell framework for golden testing of timing benchmarks.@@ -41,7 +41,8 @@         statistics >= 0.16 && < 0.17,         vector >= 0.12 && < 0.14,         boxes >= 0.1 && < 0.2,-        microlens >= 0.4 && < 0.6+        microlens >= 0.4 && < 0.6,+        deepseq >= 1.4 && < 2      hs-source-dirs:   src     default-language: Haskell2010@@ -57,3 +58,19 @@         base >= 4.14 && < 5,         golds-gym,         hspec >= 2.10 && < 3++test-suite golds-gym-properties+    type:             exitcode-stdio-1.0+    main-is:          Properties.hs+    hs-source-dirs:   test+    default-language: Haskell2010+    ghc-options:      -Wall+    build-depends:+        base >= 4.14 && < 5,+        golds-gym,+        hspec >= 2.10 && < 3,+        QuickCheck >= 2.14 && < 3,+        vector >= 0.12 && < 0.14,+        statistics >= 0.16 && < 0.17,+        text >= 1.2 && < 3,+        time >= 1.9 && < 2

@@ -26,14 +26,68 @@ -- @ -- import Test.Hspec -- import Test.Hspec.BenchGolden+-- import Data.List (sort) -- -- main :: IO () -- main = hspec $ do --   describe \"Performance\" $ do---     `benchGolden` "my algorithm" $---       return $ myAlgorithm input+--     -- Pure function with normal form evaluation+--     `benchGolden` "list sorting" $+--       `nf` sort [1000, 999..1]+--+--     -- Weak head normal form (lazy evaluation)+--     `benchGolden` "replicate" $+--       `whnf` (replicate 1000) 42+--+--     -- IO action with result forced to normal form+--     `benchGolden` "file read" $+--       `nfIO` (readFile "data.txt") -- @ --+-- __Evaluation strategies__ control how values are forced:+--+-- * 'nf' - Force to normal form (deep evaluation, use for most cases)+-- * 'whnf' - Force to weak head normal form (only outermost constructor is evaluated)+-- * 'nfIO', 'whnfIO' - Variants for IO actions+-- * 'nfAppIO', 'whnfAppIO' - For functions returning IO+-- * 'io' - Plain IO action without forcing+--+-- Without proper evaluation strategies, GHC may optimize away computations+-- or share results across iterations, making benchmarks meaningless.+--+-- = Best Practices: Avoiding Shared Thunks+--+-- __CRITICAL:__ When benchmarking with data structures, ensure the data is+-- reconstructed on each iteration to avoid measuring shared, cached results.+--+-- ❌ __Anti-pattern__ (shared list across iterations):+--+-- @+-- benchGolden "sum" $ nf sum [1..1000000]+-- @+--+-- The list @[1..1000000]@ is constructed once and shared across all iterations.+-- This allocates the entire list in memory, creates GC pressure, and prevents+-- list fusion. The first iteration evaluates the shared thunk, and subsequent+-- iterations measure cached results.+--+-- ✅ __Correct pattern__ (list reconstructed per iteration):+--+-- @+-- benchGolden "sum" $ nf (\\n -> sum [1..n]) 1000000+-- @+--+-- The lambda wrapper ensures the list is reconstructed on every iteration,+-- measuring the true cost of both construction and computation.+--+-- __Other considerations:__+--+-- * Ensure return types are inhabited enough to depend on all computations+--   (avoid @b ~ ()@ where GHC might optimize away the payload)+-- * For inlinable functions, ensure full saturation: prefer @nf (\\n -> f n) x@+--   over @nf f x@ to guarantee inlining and rewrite rules fire+-- * Use 'NFData' constraints where applicable to ensure deep evaluation+-- -- = How It Works -- -- 1. On first run, the benchmark is executed and results are saved to a@@ -103,19 +157,19 @@ -- -- Median-based comparison instead of mean -- benchGoldenWithExpectation "median test" defaultBenchConfig --   [expect _statsMedian (Percent 10.0)]---   myAction+--   (nf myAlgorithm input) -- -- -- Compose multiple expectations -- benchGoldenWithExpectation "strict test" defaultBenchConfig --   [ expect _statsMean (Percent 15.0) &&~ --     expect _statsMAD (Percent 50.0) --   ]---   myAction+--   (nf criticalFunction data) -- -- -- Expect improvement (must be faster) -- benchGoldenWithExpectation "optimization" defaultBenchConfig --   [expect _statsMean (MustImprove 10.0)]  -- Must be ≥10% faster---   myAction+--   (nf optimizedVersion input) -- @ ---- = Environment Variables --@@ -135,11 +189,21 @@      -- * Types   , BenchGolden(..)+  , BenchAction(..)   , GoldenStats(..)   , BenchResult(..)   , Warning(..)   , ArchConfig(..) +    -- * Benchmarkable Constructors+  , nf+  , whnf+  , nfIO+  , whnfIO+  , nfAppIO+  , whnfAppIO+  , io+     -- * Low-Level API   , runBenchGolden @@ -162,7 +226,7 @@ import Test.Hspec.BenchGolden.Arch import qualified Test.Hspec.BenchGolden.Lenses as L import Test.Hspec.BenchGolden.Lenses hiding (Expectation)-import Test.Hspec.BenchGolden.Runner (runBenchGolden, setAcceptGoldens, setSkipBenchmarks)+import Test.Hspec.BenchGolden.Runner (runBenchGolden, setAcceptGoldens, setSkipBenchmarks, nf, whnf, nfIO, whnfIO, nfAppIO, whnfAppIO, io) import Test.Hspec.BenchGolden.Types  -- | Create a benchmark golden test with default configuration.@@ -172,9 +236,19 @@ -- @ -- describe "Sorting" $ do --   benchGolden "quicksort 1000 elements" $---     return $ quicksort [1000, 999..1]+--     nf quicksort [1000, 999..1] -- @ --+-- Use evaluation strategy combinators to control how values are forced:+--+-- * 'nf' - Normal form (deep evaluation)+-- * 'whnf' - Weak head normal form (shallow evaluation)+-- * 'nfIO' - Normal form for IO actions+-- * 'whnfIO' - WHNF for IO actions+-- * 'nfAppIO' - Normal form for functions returning IO+-- * 'whnfAppIO' - WHNF for functions returning IO+-- * 'io' - Plain IO action (for backward compatibility)+-- -- Default configuration: -- -- * 100 iterations@@ -184,7 +258,7 @@ -- * Standard statistics (not robust mode) benchGolden ::      String  -- ^ Name of the benchmark-    -> IO () -- ^ The IO action to benchmark+    -> BenchAction -- ^ The benchmarkable action     -> Spec benchGolden name action = benchGoldenWith defaultBenchConfig name action @@ -200,7 +274,7 @@ --   , warmupIterations = 20 --   } --   "hot loop" $---   return $ criticalFunction input+--   nf criticalFunction input -- -- -- Robust statistics mode for noisy environments -- benchGoldenWith defaultBenchConfig@@ -209,11 +283,11 @@ --   , outlierThreshold = 3.0 --   } --   "benchmark with outliers" $---   return $ computation input+--   whnf computation input -- @ benchGoldenWith :: BenchConfig  -- ^ Configuration parameters     -> String -- ^ Name of the benchmark-    -> IO () -- ^ The IO action to benchmark+    -> BenchAction -- ^ The benchmarkable action     -> Spec benchGoldenWith config name action =   it name $ BenchGolden@@ -235,38 +309,38 @@ -- -- Median-based comparison (more robust to outliers) -- benchGoldenWithExpectation "median test" defaultBenchConfig --   [`expect` `_statsMedian` (`Percent` 10.0)]---   myAction+--   (nf sort [1000, 999..1]) -- -- -- Multiple metrics must pass (AND composition) -- benchGoldenWithExpectation "strict test" defaultBenchConfig --   [ expect `_statsMean` (Percent 15.0) &&~ --     expect `_statsMAD` (Percent 50.0) --   ]---   myAction+--   (nf algorithm data) -- -- -- Either metric can pass (OR composition) -- benchGoldenWithExpectation "flexible test" defaultBenchConfig --   [ expect _statsMedian (Percent 10.0) ||~ --     expect _statsMin (`Absolute` 0.01) --   ]---   myAction+--   (nf fastOp input) -- -- -- Expect performance improvement (must be faster) -- benchGoldenWithExpectation "optimization" defaultBenchConfig --   [expect _statsMean (`MustImprove` 10.0)]  -- Must be ≥10% faster---   myAction+--   (nf optimizedVersion data) -- -- -- Expect controlled regression (for feature additions) -- benchGoldenWithExpectation "new feature" defaultBenchConfig --   [expect _statsMean (`MustRegress` 5.0)]  -- Accept 5-20% slowdown---   myAction+--   (nf newFeature input) -- -- -- Low variance requirement -- benchGoldenWithExpectation "stable perf" defaultBenchConfig --   [ expect _statsMean (Percent 15.0) &&~ --     expect `_statsIQR` (Absolute 0.1) --   ]---   myAction+--   (nfIO stableOperation) -- @ -- -- Note: Expectations are checked against golden files. On first run, a baseline@@ -275,7 +349,7 @@     String        -- ^ Name of the benchmark     -> BenchConfig  -- ^ Configuration parameters     -> [L.Expectation]  -- ^ List of expectations (all must pass)-    -> IO ()       -- ^ The IO action to benchmark+    -> BenchAction       -- ^ The benchmarkable action     -> Spec benchGoldenWithExpectation name config expectations action =   it name $ BenchGoldenWithExpectations name action config expectations@@ -283,7 +357,7 @@ -- | Data type for benchmarks with custom lens-based expectations. data BenchGoldenWithExpectations = BenchGoldenWithExpectations   !String        -- Name-  !(IO ())       -- Action+  !BenchAction   -- Action   !BenchConfig   -- Config   ![L.Expectation] -- Expectations @@ -355,12 +429,17 @@     readIORef ref  -- | Run a benchmark with custom expectations.-runBenchGoldenWithExpectations :: String -> IO () -> BenchConfig -> [L.Expectation] -> IO BenchResult+runBenchGoldenWithExpectations :: String -> BenchAction -> BenchConfig -> [L.Expectation] -> IO BenchResult runBenchGoldenWithExpectations name action config expectations = do   -- Convert to BenchGolden and run normally first   let bg = BenchGolden name action config   result <- runBenchGolden bg   +  -- Extract tolerance from first expectation for error messages+  let (tolPct, tolAbs) = case expectations of+        [] -> (tolerancePercent config, absoluteToleranceMs config)+        (e:_) -> L.toleranceFromExpectation e+     -- Then check expectations for Pass/Regression/Improvement results   case result of     FirstRun stats -> return $ FirstRun stats@@ -377,19 +456,19 @@                meanDiff = if goldenVal == 0                           then 100.0                           else ((actualVal - goldenVal) / goldenVal) * 100-           in return $ Regression golden actual meanDiff (tolerancePercent config) (absoluteToleranceMs config)-    Regression golden actual pct tol absTol ->+           in return $ Regression golden actual meanDiff tolPct tolAbs+    Regression golden actual pct _tol _absTol ->       -- Check if regression is acceptable per expectations       let allPass = all (\e -> L.checkExpectation e golden actual) expectations       in if allPass          then return $ Pass golden actual []-         else return $ Regression golden actual pct tol absTol-    Improvement golden actual pct tol absTol ->+         else return $ Regression golden actual pct tolPct tolAbs+    Improvement golden actual pct _tol _absTol ->       -- Check if improvement satisfies expectations       let allPass = all (\e -> L.checkExpectation e golden actual) expectations       in if allPass          then return $ Pass golden actual []-         else return $ Improvement golden actual pct tol absTol+         else return $ Improvement golden actual pct tolPct tolAbs  -- | Convert expectation-based benchmark result to hspec Result. fromBenchResultWithExpectations :: [L.Expectation] -> BenchResult -> Result@@ -411,8 +490,11 @@         toleranceDesc = case absToleranceMs of           Nothing -> printf "tolerance: %.1f%%" tolerance           Just absMs -> printf "tolerance: %.1f%% or %.3f ms" tolerance absMs-        message = printf "Mean time increased by %.1f%% (%s)\n\n%s"-                    pctChange toleranceDesc (formatRegression golden actual)+        changeVerb :: String+        changeVerb = if pctChange >= 0 then "increased" else "decreased"+        absPctChange = abs pctChange+        message = printf "Mean time %s by %.1f%% (%s)\n\n%s"+                    changeVerb absPctChange toleranceDesc (formatRegression golden actual)     in Result message (Failure Nothing (Reason message))    Improvement golden actual pctChange tolerance absToleranceMs ->@@ -444,15 +526,6 @@       -- Create detailed comparison table       metricCol = Box.vcat Box.left $ map Box.text          ["Metric", "------", "Mean", "Stddev", "Median", "Min", "Max"]-      actualCol = Box.vcat Box.right $ map Box.text -        [ "Actual"-        , "------"-        , printf "%.3f ms" (statsMean actual)-        , printf "%.3f ms" (statsStddev actual)-        , printf "%.3f ms" (statsMedian actual)-        , printf "%.3f ms" (statsMin actual)-        , printf "%.3f ms" (statsMax actual)-        ]       baselineCol = Box.vcat Box.right $ map Box.text         [ "Baseline"         , "--------"@@ -462,6 +535,15 @@         , printf "%.3f ms" (statsMin golden)         , printf "%.3f ms" (statsMax golden)         ]+      actualCol = Box.vcat Box.right $ map Box.text +        [ "Actual"+        , "------"+        , printf "%.3f ms" (statsMean actual)+        , printf "%.3f ms" (statsStddev actual)+        , printf "%.3f ms" (statsMedian actual)+        , printf "%.3f ms" (statsMin actual)+        , printf "%.3f ms" (statsMax actual)+        ]       diffCol = Box.vcat Box.right $ map Box.text         [ "Diff"         , "----"@@ -472,7 +554,7 @@         , ""         ]       -      table = Box.hsep 2 Box.top [metricCol, actualCol, baselineCol, diffCol]+      table = Box.hsep 2 Box.top [metricCol, baselineCol, actualCol, diffCol]   in Box.render table  -- | Format a passing comparison.@@ -485,20 +567,20 @@                    then 0                    else ((statsStddev actual - statsStddev golden) / statsStddev golden) * 100       -      -- Create table with metric, actual, baseline, and diff columns+      -- Create table with metric, baseline, actual, and diff columns       metricCol = Box.vcat Box.left $ map Box.text ["Metric", "------", "Mean", "Stddev"]-      actualCol = Box.vcat Box.right $ map Box.text -        [ "Actual"-        , "------"-        , printf "%.3f ms" (statsMean actual)-        , printf "%.3f ms" (statsStddev actual)-        ]       baselineCol = Box.vcat Box.right $ map Box.text         [ "Baseline"         , "--------"         , printf "%.3f ms" (statsMean golden)         , printf "%.3f ms" (statsStddev golden)         ]+      actualCol = Box.vcat Box.right $ map Box.text +        [ "Actual"+        , "------"+        , printf "%.3f ms" (statsMean actual)+        , printf "%.3f ms" (statsStddev actual)+        ]       diffCol = Box.vcat Box.right $ map Box.text         [ "Diff"         , "----"@@ -506,7 +588,7 @@         , printf "%+.1f%%" stddevDiff         ]       -      table = Box.hsep 2 Box.top [metricCol, actualCol, baselineCol, diffCol]+      table = Box.hsep 2 Box.top [metricCol, baselineCol, actualCol, diffCol]   in Box.render table  -- | Format statistics for display.

@@ -7,7 +7,7 @@ -- Description : Architecture detection for machine-specific golden files -- Copyright   : (c) 2026 -- License     : MIT--- Maintainer  : your.email@example.com+-- Maintainer  : @ocramz -- -- This module provides functions for detecting the current machine's -- architecture, which is used to create architecture-specific golden files.

@@ -6,7 +6,7 @@ -- Description : Lens-based expectation combinators for benchmark comparison -- Copyright   : (c) 2026 -- License     : MIT--- Maintainer  : your.email@example.com+-- Maintainer  : @ocramz -- -- This module provides van Laarhoven lenses for 'GoldenStats' fields and -- expectation combinators for building custom performance assertions.@@ -124,6 +124,8 @@     -- * Utilities   , percentDiff   , absDiff+  , toleranceFromExpectation+  , toleranceValues   ) where  import Lens.Micro@@ -444,3 +446,18 @@ -- Returns: @abs(actual - baseline)@ absDiff :: Double -> Double -> Double absDiff baseline actual = abs (actual - baseline)++-- | Extract tolerance description from an expectation for error messages.+-- For compound expectations (And/Or), returns the first tolerance found.+toleranceFromExpectation :: Expectation -> (Double, Maybe Double)+toleranceFromExpectation (ExpectStat _ tol) = toleranceValues tol+toleranceFromExpectation (And e1 _) = toleranceFromExpectation e1+toleranceFromExpectation (Or e1 _) = toleranceFromExpectation e1++-- | Extract percentage and optional absolute tolerance from a Tolerance.+toleranceValues :: Tolerance -> (Double, Maybe Double)+toleranceValues (Percent pct) = (pct, Nothing)+toleranceValues (Absolute abs_) = (0, Just abs_)+toleranceValues (Hybrid pct abs_) = (pct, Just abs_)+toleranceValues (MustImprove pct) = (pct, Nothing)+toleranceValues (MustRegress pct) = (pct, Nothing)

@@ -1,12 +1,15 @@+{-# LANGUAGE BangPatterns #-}+{-# LANGUAGE MagicHash #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RecordWildCards #-}+{-# LANGUAGE ScopedTypeVariables #-}  -- | -- Module      : Test.Hspec.BenchGolden.Runner -- Description : Benchmark execution and golden file comparison -- Copyright   : (c) 2026 -- License     : MIT--- Maintainer  : your.email@example.com+-- Maintainer  : @ocramz -- -- This module handles running benchmarks and comparing results against -- golden files. It includes:@@ -15,6 +18,22 @@ -- * Golden file I/O (reading/writing JSON statistics) -- * Tolerance-based comparison with variance warnings -- * Support for updating baselines via GOLDS_GYM_ACCEPT environment variable+-- * Evaluation strategies to control how values are forced (nf, whnf, etc.)+--+-- = Evaluation Strategies+--+-- Benchmarks require explicit evaluation strategies to prevent GHC from+-- optimizing away computations or sharing results across iterations:+--+-- * 'nf' - Force result to normal form (deep, full evaluation)+-- * 'whnf' - Force result to weak head normal form (shallow evaluation)+-- * 'nfIO' - Execute IO and force result to normal form+-- * 'whnfIO' - Execute IO and force result to WHNF+-- * 'nfAppIO' - Apply function, execute IO, force result to normal form+-- * 'whnfAppIO' - Apply function, execute IO, force result to WHNF+-- * 'io' - Plain IO without additional forcing+--+-- These are vendored from tasty-bench with proper attribution (BSD-3-Clause).  module Test.Hspec.BenchGolden.Runner   ( -- * Running Benchmarks@@ -45,8 +64,19 @@   , shouldSkipBenchmarks   , setAcceptGoldens   , setSkipBenchmarks++    -- * Benchmarkable Constructors+  , io+  , nf+  , whnf+  , nfIO+  , whnfIO+  , nfAppIO+  , whnfAppIO   ) where +import Control.DeepSeq (NFData, rnf)+import Control.Exception (evaluate) import Control.Monad (when, replicateM_) import Data.Aeson (eitherDecodeFileStrict, encodeFile) import Data.IORef (IORef, newIORef, readIORef, writeIORef)@@ -54,11 +84,13 @@ import qualified Data.Text as T import Data.Time (getCurrentTime) import qualified Data.Vector.Unboxed as V+import Data.Word (Word64) import qualified Statistics.Sample as Stats import System.CPUTime (getCPUTime) import System.Directory (createDirectoryIfMissing, doesFileExist) import System.FilePath ((</>), (<.>)) import System.IO.Unsafe (unsafePerformIO)+import GHC.Exts (SPEC(..))  import Lens.Micro ((^.)) @@ -68,6 +100,138 @@ import Test.Hspec.BenchGolden.Lenses (metricFor, varianceFor) import Test.Hspec.BenchGolden.Types +-- -----------------------------------------------------------------------------+-- Evaluation Strategies+-- Vendored from tasty-bench-0.5 with modifications+-- Copyright (c) 2021 Andrew Lelechenko and tasty-bench contributors+-- MIT License+-- https://hackage.haskell.org/package/tasty-bench-0.5+-- -----------------------------------------------------------------------------++-- | Benchmark a pure function applied to an argument, forcing the result to+-- normal form (NF) using 'rnf' from "Control.DeepSeq".+-- This ensures the entire result structure is evaluated.+--+-- Example:+-- @+-- benchGolden "fib 30" (nf fib 30)+-- @+nf :: NFData b => (a -> b) -> a -> BenchAction+nf = funcToBench rnf+{-# INLINE nf #-}++-- | Benchmark a pure function applied to an argument, forcing the result to+-- weak head normal form (WHNF) only. This evaluates just the outermost constructor.+--+-- Example:+-- @+-- benchGolden "replicate" (whnf (replicate 1000) 42)+-- @+whnf :: (a -> b) -> a -> BenchAction+whnf = funcToBench id+{-# INLINE whnf #-}++-- | Benchmark an 'IO' action, forcing the result to normal form.+--+-- Example:+-- @+-- benchGolden "readFile" (nfIO $ readFile "data.txt")+-- @+nfIO :: NFData a => IO a -> BenchAction+nfIO = ioToBench rnf+{-# INLINE nfIO #-}++-- | Benchmark an 'IO' action, forcing the result to weak head normal form.+--+-- Example:+-- @+-- benchGolden "getLine" (whnfIO getLine)+-- @+whnfIO :: IO a -> BenchAction+whnfIO = ioToBench id+{-# INLINE whnfIO #-}++-- | Benchmark a function that performs 'IO', forcing the result to normal form.+--+-- Example:+-- @+-- benchGolden "lookup in map" (nfAppIO lookupInDB "key")+-- @+nfAppIO :: NFData b => (a -> IO b) -> a -> BenchAction+nfAppIO = ioFuncToBench rnf+{-# INLINE nfAppIO #-}++-- | Benchmark a function that performs 'IO', forcing the result to weak head normal form.+--+-- Example:+-- @+-- benchGolden "query database" (whnfAppIO queryDB params)+-- @+whnfAppIO :: (a -> IO b) -> a -> BenchAction+whnfAppIO = ioFuncToBench id+{-# INLINE whnfAppIO #-}++-- | Benchmark an 'IO' action, discarding the result.+-- This is for backward compatibility with code that uses @IO ()@ actions.+--+-- Example:+-- @+-- benchGolden "compute" (io $ do+--   result <- heavyComputation+--   evaluate result)+-- @+io :: IO () -> BenchAction+io action = BenchAction (\n -> replicateM_ (fromIntegral n) action)+{-# INLINE io #-}++-- Internal helpers++funcToBench :: forall a b c. (b -> c) -> (a -> b) -> a -> BenchAction+funcToBench frc = (BenchAction .) . funcToBenchLoop SPEC+  where+    -- Here we rely on the fact that GHC (unless spurred by+    -- -fstatic-argument-transformation) is not smart enough:+    -- it does not notice that `f` and `x` arguments are loop invariant+    -- and could be floated, and the whole `f x` expression shared.+    -- If we create a closure with `f` and `x` bound in the environment,+    -- then GHC is smart enough to share computation of `f x`.+    funcToBenchLoop :: SPEC -> (a -> b) -> a -> Word64 -> IO ()+    funcToBenchLoop !_ f x n+      | n == 0    = pure ()+      | otherwise = do+        _ <- evaluate (frc (f x))+        funcToBenchLoop SPEC f x (n - 1)+{-# INLINE funcToBench #-}++ioToBench :: forall a b. (a -> b) -> IO a -> BenchAction+ioToBench frc act = BenchAction (ioToBenchLoop SPEC act)+  where+    ioToBenchLoop :: SPEC -> IO a -> Word64 -> IO ()+    ioToBenchLoop !_ action n+      | n == 0    = pure ()+      | otherwise = do+        x <- action+        _ <- evaluate (frc x)+        ioToBenchLoop SPEC action (n - 1)+{-# INLINE ioToBench #-}++ioFuncToBench :: forall a b c. (b -> c) -> (a -> IO b) -> a -> BenchAction+ioFuncToBench frc = (BenchAction .) . ioFuncToBenchLoop SPEC+  where+    ioFuncToBenchLoop :: SPEC -> (a -> IO b) -> a -> Word64 -> IO ()+    ioFuncToBenchLoop !_ f x n+      | n == 0    = pure ()+      | otherwise = do+        y <- f x+        _ <- evaluate (frc y)+        ioFuncToBenchLoop SPEC f x (n - 1)+{-# INLINE ioFuncToBench #-}++-- -----------------------------------------------------------------------------+-- Benchmark execution+-- -----------------------------------------------------------------------------++ -- | Run a benchmark golden test. -- -- This function:@@ -102,7 +266,7 @@        -- Run warm-up iterations       when (warmupIterations config > 0) $-        replicateM_ (warmupIterations config) benchAction+        runBenchAction benchAction (fromIntegral $ warmupIterations config)        -- Run the actual benchmark       actualStats <- runBenchmark benchName benchAction config arch@@ -132,7 +296,7 @@               return result  -- | Run a benchmark and collect statistics.-runBenchmark :: String -> IO () -> BenchConfig -> ArchConfig -> IO GoldenStats+runBenchmark :: String -> BenchAction -> BenchConfig -> ArchConfig -> IO GoldenStats runBenchmark _name action config arch = do   if useRobustStatistics config     then runBenchmarkWithRawTimings _name action config arch@@ -141,7 +305,7 @@       (cpuStats, _wallStats) <- BP.benchmark         (iterations config)         (pure ())                    -- setup-        (const action)               -- action+        (const $ runBenchAction action 1)  -- action: run 1 iteration         (const $ pure ())            -- teardown        now <- getCurrentTime@@ -162,10 +326,10 @@         }  -- | Run a benchmark with raw timing collection for robust statistics.-runBenchmarkWithRawTimings :: String -> IO () -> BenchConfig -> ArchConfig -> IO GoldenStats+runBenchmarkWithRawTimings :: String -> BenchAction -> BenchConfig -> ArchConfig -> IO GoldenStats runBenchmarkWithRawTimings _name action config arch = do   -- Collect raw CPU timings-  timings <- mapM (const measureCPUTime) [1 .. iterations config]+  timings <- mapM (const $ measureCPUTimeForAction action) [1 .. iterations config]      let sortedTimings = sort timings       vec = V.fromList sortedTimings@@ -203,9 +367,9 @@     , statsOutliers    = outliers'     }   where-    measureCPUTime = do+    measureCPUTimeForAction act = do       startCpu <- getCPUTime-      action+      runBenchAction act 1  -- Run the action once       endCpu <- getCPUTime       let cpuTime = picosToMillis (endCpu - startCpu)       return cpuTime

@@ -7,7 +7,7 @@ -- Description : Core types for benchmark golden testing -- Copyright   : (c) 2026 -- License     : MIT--- Maintainer  : your.email@example.com+-- Maintainer  : @ocramz -- -- This module defines the core data types used by the golds-gym framework: --@@ -22,6 +22,7 @@     BenchGolden(..)   , BenchConfig(..)   , defaultBenchConfig+  , BenchAction(..)      -- * Golden File Statistics   , GoldenStats(..)@@ -37,16 +38,21 @@ import Data.Aeson import Data.Text (Text) import Data.Time (UTCTime)+import Data.Word (Word64) import GHC.Generics (Generic)  -- Note: Expectation type is defined in Lenses module to avoid circular imports +-- | A benchmarkable action that can be run multiple times.+-- The 'Word64' parameter represents the number of iterations to execute.+newtype BenchAction = BenchAction { runBenchAction :: Word64 -> IO () }+ -- | Configuration for a single benchmark golden test. data BenchGolden = BenchGolden   { benchName   :: !String     -- ^ Name of the benchmark (used for golden file naming)-  , benchAction :: !(IO ())-    -- ^ The IO action to benchmark+  , benchAction :: !BenchAction+    -- ^ The benchmarkable action to run   , benchConfig :: !BenchConfig     -- ^ Configuration parameters   }

@@ -0,0 +1,312 @@+{-# LANGUAGE ScopedTypeVariables #-}++-- | Property-based tests for statistical computations in Runner module+module Main (main) where++import Data.List (sort)+import qualified Data.Text as T+import qualified Data.Vector.Unboxed as V+import qualified Statistics.Sample as Stats+import Test.Hspec+import Test.QuickCheck++import Test.Hspec.BenchGolden.Runner+import Test.Hspec.BenchGolden.Types++main :: IO ()+main = hspec spec++spec :: Spec+spec = do+  describe "calculateTrimmedMean" $ do+    it "returns value within min and max bounds for non-empty vectors" $ property $+      \(PositiveVector vec) (ValidTrimPercent pct) ->+        V.length vec > 0 ==>+        let result = calculateTrimmedMean pct vec+            minVal = V.minimum vec+            maxVal = V.maximum vec+        in result >= minVal && result <= maxVal++    it "converges to regular mean when trim percentage is zero" $ property $+      \(PositiveVector vec) ->+        not (V.null vec) ==>+        let trimmed = calculateTrimmedMean 0.0 vec+            regular = Stats.mean vec+        in abs (trimmed - regular) < 1e-10++    it "falls back to regular mean for trim percentage >= 50" $ property $+      \(PositiveVector vec) (Positive pct) ->+        V.length vec > 0 ==>+        let trimPct = 50.0 + pct+            trimmed = calculateTrimmedMean trimPct vec+            regular = Stats.mean vec+        in abs (trimmed - regular) < 1e-10++    it "returns zero for empty vector" $+      calculateTrimmedMean 10.0 V.empty `shouldBe` 0.0++    it "is less affected by outliers than regular mean" $ property $+      \(PositiveVector vec) ->+        V.length vec > 4 ==>+        let withOutlier = V.snoc vec (V.maximum vec * 100)  -- Add extreme outlier+            trimmed = calculateTrimmedMean 10.0 withOutlier+            regular = Stats.mean withOutlier+            originalMean = Stats.mean vec+        in abs (trimmed - originalMean) < abs (regular - originalMean)++  describe "calculateMAD" $ do+    it "is always non-negative" $ property $+      \(PositiveVector vec) ->+        let median' = if V.null vec then 0 else V.unsafeIndex vec (V.length vec `div` 2)+            mad = calculateMAD vec median'+        in mad >= 0++    it "returns zero for empty vector" $+      calculateMAD V.empty 0.0 `shouldBe` 0.0++    it "returns zero for constant data" $ property $+      \(Positive (constant :: Double)) (Positive (n :: Int)) ->+        n < 100 ==>+        let vec = V.replicate n constant+            median' = constant+            mad = calculateMAD vec median'+        in mad == 0.0++    it "scales linearly with data (scale invariance)" $ property $+      \(PositiveVector vec) (Positive s) ->+        V.length vec > 0 ==>+        let median' = V.unsafeIndex vec (V.length vec `div` 2)+            mad1 = calculateMAD vec median'+            scaledVec = V.map (* s) vec+            scaledMedian = median' * s+            mad2 = calculateMAD scaledVec scaledMedian+        in abs (mad2 - mad1 * s) < 1e-6++    -- Note: The relationship MAD ≤ σ only holds for normal distributions.+    -- For arbitrary distributions (bimodal, skewed, etc.), MAD can exceed σ.+    -- Therefore we don't test this as a universal property.++  describe "calculateIQR" $ do+    it "is always non-negative" $ property $+      \(PositiveVector vec) ->+        calculateIQR vec >= 0++    it "returns zero for vectors with less than 4 elements" $+      let vec1 = V.fromList [1.0]+          vec2 = V.fromList [1.0, 2.0]+          vec3 = V.fromList [1.0, 2.0, 3.0]+      in calculateIQR vec1 == 0.0 &&+         calculateIQR vec2 == 0.0 &&+         calculateIQR vec3 == 0.0++    it "returns zero for constant data" $ property $+      \(Positive (constant :: Double)) (Positive (n :: Int)) ->+        n < 100 && n >= 4 ==>+        let vec = V.replicate n constant+            iqr = calculateIQR vec+        in iqr == 0.0++    it "approximates 1.349 * stddev for normal-ish distributions" $ property $+      \(PositiveVector vec) ->+        V.length vec >= 20 ==>  -- Need sufficient sample size+        let iqr = calculateIQR vec+            stddev = Stats.stdDev vec+            ratio = if stddev == 0 then 0 else iqr / stddev+        in ratio >= 0.5 && ratio <= 2.5  -- Loose bounds for non-normal data++  describe "detectOutliers" $ do+    it "returns empty list for empty vector" $+      detectOutliers 3.0 V.empty 0.0 0.0 `shouldBe` []++    it "returns empty list when MAD is zero" $ property $+      \(Positive (constant :: Double)) (Positive (n :: Int)) (Positive thresh) ->+        n < 100 ==>+        let vec = V.replicate n constant+            outliers = detectOutliers thresh vec constant 0.0+        in null outliers++    it "fewer outliers with higher threshold (monotonicity)" $ property $+      \(PositiveVector vec) (Positive thresh1) (Positive thresh2) ->+        not (V.null vec) && thresh1 < thresh2 ==>+        let median' = V.unsafeIndex vec (V.length vec `div` 2)+            mad = calculateMAD vec median'+            outliers1 = detectOutliers thresh1 vec median' mad+            outliers2 = detectOutliers thresh2 vec median' mad+        in length outliers1 >= length outliers2++    it "all outliers satisfy distance criterion" $ property $+      \(PositiveVector vec) (Positive thresh) ->+        not (V.null vec) ==>+        let median' = V.unsafeIndex vec (V.length vec `div` 2)+            mad = calculateMAD vec median'+            outliers = detectOutliers thresh vec median' mad+        in all (\x -> abs (x - median') > thresh * mad || mad == 0) outliers++    it "all outliers are from original vector" $ property $+      \(PositiveVector vec) (Positive thresh) ->+        not (V.null vec) ==>+        let median' = V.unsafeIndex vec (V.length vec `div` 2)+            mad = calculateMAD vec median'+            outliers = detectOutliers thresh vec median' mad+            vecList = V.toList vec+        in all (`elem` vecList) outliers++  describe "calculateRobustStats" $ do+    it "returns consistent tuple structure" $ property $+      \(PositiveVector vec) ->+        let config = defaultBenchConfig { useRobustStatistics = True }+            median' = if V.null vec then 0 else V.unsafeIndex vec (V.length vec `div` 2)+            (tm, mad, iqr, outliers) = calculateRobustStats config vec median'+        in tm >= 0 && mad >= 0 && iqr >= 0 && length outliers >= 0++  describe "compareStats (tolerance logic)" $ do+    it "self-comparison always passes" $ property $+      \(SmallPositive mean') (SmallPositive stddev') ->+        let stats = makeGoldenStats mean' stddev'+            config = defaultBenchConfig { useRobustStatistics = False }+            result = compareStats config stats stats+        in case result of+             Pass _ _ _ -> True+             _          -> False++    it "passes if within percentage tolerance" $ property $+      \(SmallPositive golden) ->+        golden > 0.01 ==>  -- Avoid very small baselines+        let delta = golden * 0.08  -- 8% change, well within 15%+            goldenStats = makeGoldenStats golden 0.1+            actualStats = makeGoldenStats (golden + delta) 0.1+            config = defaultBenchConfig { tolerancePercent = 15.0, absoluteToleranceMs = Nothing }+            result = compareStats config goldenStats actualStats+        in case result of+             Pass _ _ _ -> True+             _          -> False++    it "passes if within absolute tolerance even with high percentage" $+      let golden = 0.002  -- 2 microseconds - very small+          actual = 0.004  -- 4 microseconds - 100% increase+          goldenStats = makeGoldenStats golden 0.0001+          actualStats = makeGoldenStats actual 0.0001+          config = defaultBenchConfig { absoluteToleranceMs = Just 0.01 }  -- 10 microseconds+          result = compareStats config goldenStats actualStats+      in case result of+           Pass _ _ _ -> True+           _          -> False++    it "regresses when exceeding both tolerances" $ property $+      \(SmallPositive golden) ->+        golden > 0.1 ==>  -- Large enough baseline+        let goldenStats = makeGoldenStats golden 0.01+            actualStats = makeGoldenStats (golden * 2) 0.01  -- 100% increase+            config = defaultBenchConfig { tolerancePercent = 10.0, absoluteToleranceMs = Just 0.01 }+            result = compareStats config goldenStats actualStats+        in case result of+             Regression _ _ _ _ _ -> True+             _                    -> False++    it "handles zero golden value gracefully" $+      let goldenStats = makeGoldenStats 0.0 0.0+          actualStats = makeGoldenStats 1.0 0.1+          config = defaultBenchConfig+          result = compareStats config goldenStats actualStats+      in case result of+           Regression _ _ pct _ _ -> pct == 100.0+           _                      -> False++  describe "checkVariance" $ do+    it "returns only variance warnings when enabled" $ property $+      \(SmallPositive m1) (SmallPositive m2) (SmallPositive s1) (SmallPositive s2) ->+        let config = defaultBenchConfig { warnOnVarianceChange = True, useRobustStatistics = False }+            golden = makeGoldenStats m1 s1+            actual = makeGoldenStats m2 s2+            warnings = checkVariance config golden actual+            -- Filter to only variance-related warnings (not outliers)+            varianceWarnings = filter isVarianceWarning warnings+        in varianceWarnings == warnings  -- All warnings should be variance-related++    it "detects high coefficient of variation" $ property $+      \(SmallPositive mean') ->+        mean' > 0.01 ==>+        let stddev' = mean' * 2  -- CV = 200%+            config = defaultBenchConfig { warnOnVarianceChange = True }+            stats = makeGoldenStats mean' stddev'+            warnings = checkVariance config stats stats+        in any isHighVarianceWarning warnings++    it "detects variance increase beyond tolerance" $ property $+      \(SmallPositive mean') (SmallPositive s1) ->+        s1 > 0 ==>+        let s2 = s1 * 3  -- 200% increase+            config = defaultBenchConfig { warnOnVarianceChange = True, varianceTolerancePercent = 50.0 }+            golden = makeGoldenStats mean' s1+            actual = makeGoldenStats mean' s2+            warnings = checkVariance config golden actual+        in any isVarianceIncreasedWarning warnings++-- -----------------------------------------------------------------------------+-- Custom Generators and Helper Types+-- -----------------------------------------------------------------------------++-- | Vector of positive doubles (simulating timing data)+newtype PositiveVector = PositiveVector (V.Vector Double)+  deriving (Show)++instance Arbitrary PositiveVector where+  arbitrary = do+    n <- choose (0, 50)  -- Reasonable vector size+    values <- vectorOf n (choose (0.001, 100.0))  -- 1μs to 100ms range+    return $ PositiveVector $ V.fromList $ sort values+  shrink (PositiveVector vec)+    | V.null vec = []+    | otherwise = [PositiveVector (V.take (V.length vec `div` 2) vec)]++-- | Valid trim percentage (0-50)+newtype ValidTrimPercent = ValidTrimPercent Double+  deriving (Show)++instance Arbitrary ValidTrimPercent where+  arbitrary = ValidTrimPercent <$> choose (0.0, 49.9)++-- | Small positive number for timing tests+newtype SmallPositive = SmallPositive Double+  deriving (Show)++instance Arbitrary SmallPositive where+  arbitrary = SmallPositive <$> choose (0.001, 10.0)++-- -----------------------------------------------------------------------------+-- Helper Functions+-- -----------------------------------------------------------------------------++-- | Create minimal GoldenStats for testing+makeGoldenStats :: Double -> Double -> GoldenStats+makeGoldenStats mean' stddev' = GoldenStats+  { statsMean = mean'+  , statsStddev = stddev'+  , statsMedian = mean'+  , statsMin = mean' - stddev'+  , statsMax = mean' + stddev'+  , statsPercentiles = []+  , statsArch = T.pack "test-arch"+  , statsTimestamp = read "2026-01-01 00:00:00 UTC"+  , statsTrimmedMean = mean'+  , statsMAD = stddev' * 0.7  -- Approximate relationship+  , statsIQR = stddev' * 1.35+  , statsOutliers = []+  }++-- | Check if warning is HighVariance+isHighVarianceWarning :: Warning -> Bool+isHighVarianceWarning (HighVariance _) = True+isHighVarianceWarning _ = False++-- | Check if warning is VarianceIncreased+isVarianceIncreasedWarning :: Warning -> Bool+isVarianceIncreasedWarning (VarianceIncreased _ _ _ _) = True+isVarianceIncreasedWarning _ = False++-- | Check if warning is any variance-related warning+isVarianceWarning :: Warning -> Bool+isVarianceWarning (VarianceIncreased _ _ _ _) = True+isVarianceWarning (VarianceDecreased _ _ _ _) = True+isVarianceWarning (HighVariance _) = True+isVarianceWarning _ = False