diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Revision history for dataframe +## 3.7.0.0 + +* `dataframe-parquet`: a requested column is matched by its full name first, + so a top-level column named with a dot (`sepal.length`) is read and projected + as written; the last segment is only an alias for nested leaves. Fixes typed + reads of such files, which failed with "Columns not found". +* `fromNamedColumns` is now `fromColumns`. +* `HasSchema`'s methods `toColumns`/`fromColumns` are now + `recordsToColumns`/`recordsFromFrame`. +* `UExpr` is now `SomeExpr` (`toUExpr`/`fromUExpr` → + `toSomeExpr`/`fromSomeExpr`). +* `dataframe-learn`: `GBConfig` gained fields (`gbMaxBins`, `gbL2`, + `gbSubsample`, ...) and lost its `Eq` instance. +* `dataframe-parquet`: writer internals now run in `ST s` instead of `IO`. +* More ergonomic conditionals with `caseWhen`/`orElse`. +* `countDistinct`, `topBy`, and `(=:)` for building columns by hand. +* Tree learners handle nullable columns. +* Parallel parquet reader with streaming numeric decode. +* Faster parquet writer. +* Fused parallel kernels for whole-frame `sum`/`mean`, `countDistinct` and + packed-text kernels. +* `pow` and `daysBetween` no longer treat their arguments as commutative. +* Nullable int32/int64 decimal parquet columns keep their nulls. +* The parquet reader selects and filters columns whose names contain dots. +* `countRows` reads files as UTF-8. + ## 3.6.1.0 * splices for deriving schema @@ -119,7 +145,7 @@ A large performance and ML release. ### Highlights -* **Much faster I/O and analytics.** CSV reading, group-by, joins and sorting were rebuilt on compact unboxed / `PackedText` columns, parallel open-addressing hash tables, and vectorized aggregation; the default reader drops `cassava` for a single-pass scanner and `dataframe-fastcsv` adds multicore chunking. The end-to-end join + group-by pipeline runs several times faster with much lower memory and scales with `-threaded` / `+RTS -N`; results are byte-identical (golden-tested). +* **Much faster I/O and analytics.** CSV reading, group-by, joins and sorting were rebuilt on compact unboxed / `PackedText` columns, parallel open-addressing hash tables, and vectorized aggregation; the default reader drops `cassava` for a single-pass scanner and `dataframe-fastcsv` adds multicore chunking. The end-to-end join + group-by pipeline runs several times faster with much lower memory and scales with `-threaded` / `+RTS -N`. * **Lazy engine on par with eager** — bounded-source queries route through the same fast paths (streaming preserved for unbounded); lazy `sortBy` now also orders non-`Text` columns correctly. * **New ML library (`dataframe-learn`).** scikit-learn-style estimators behind a uniform `fit` / `predict`: linear / ridge / lasso / logistic regression, SVM, trees, boosting, k-means, GMM, DBSCAN, PCA and kernel PCA, symbolic regression and feature synthesis, with metrics and cross-validation. Pure and deterministic; every model also compiles to a dataframe `Expr`. * **Typed joins are checked at compile time** — keys must exist in both schemas with matching types (previously a runtime failure or silent empty result). diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -35,19 +35,33 @@ # DataFrame -Tabular data analysis in Haskell. Read CSV, Parquet, and JSON files, transform columns with a typed expression DSL, and optionally lock down your entire schema at the type level for compile-time safety. +Tabular data analysis and symbolic AI in Haskell. Supports eager computation and lazy/deferred computation. API flexible enough to accomodate beginners to both data analysis or Haskell, and advanced type-safe pipeline. -The library ships three API layers — all operating on the same underlying `DataFrame` type at runtime: +## Features -- **Untyped** (`import qualified DataFrame as D`) — string-based column names, great for exploration and scripting. -- **Typed** (`import qualified DataFrame.Typed as T`) — phantom-type schema tracking with compile-time column validation. -- **Monadic API** — write your transformation as a self contained pipeline. +**I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from Hugging Face datasets (`hf://` URIs) via the `dataframe-huggingface` package. Column projection and predicate pushdown for Parquet reads. +**Operations**: filter, select, derive, groupBy, aggregate, joins (inner, left, right, full outer), sort, sample, stratified sample, distinct, k-fold splits. + +**Expressions**: typed column references (`F.col @Double "x"`), arithmetic, comparisons, logical operators, nullable-aware three-valued logic (`.==`, `.&&`), string matching (`like`, `regex`), casting, and user-defined functions via `lift`/`lift2`. + +**Statistics**: mean, median, mode, variance, standard deviation, percentiles, inter-quartile range, correlation, skewness, frequency tables, imputation. + +**Plotting**: terminal plots (histogram, scatter, line, bar, box, pie, heatmap, stacked bar, correlation matrix) and interactive HTML plots. + +**Lazy engine**: streaming query execution for files that don't fit in memory. Rule-based optimizer with filter fusion, predicate pushdown, and dead column elimination. Pull-based executor with configurable batch sizes. + +**Interop**: Arrow C Data Interface for zero-copy round-trips with Python and Polars. + +**ML**: decision trees (TAO algorithm), feature synthesis, k-fold cross-validation, stratified sampling. + +**Notebooks**: IHaskell integration with [pre-built Binder examples](https://mybinder.org/v2/gh/mchav/ihaskell-dataframe/HEAD). + > **This README is a runnable [scripths](https://github.com/DataHaskell/scripths) notebook.** Every Haskell block runs top-to-bottom in one shared session against the datasets in [`./data`](./data). Reproduce every output below with `scripths docs/base_scripts/base_readme.md -o README.md` from the repo root. ## Why this library? -* Concise, declarative, composable data pipelines using the `|>` pipe operator. +* Concise, declarative, composable data pipelines and machine learning. * Choose your level of type safety: keep it lightweight for quick analysis, or lock it down for production pipelines. * High performance from Haskell's optimizing compiler and an efficient columnar memory model based onn Apache Arrow. * Designed for interactivity: a custom REPL, IHaskell/Sabela notebook support, terminal and web plotting, and helpful error messages. @@ -69,18 +83,13 @@ ```bash $ dataframe -dataframe> df = D.fromNamedColumns [("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int]), ("amount", D.fromList [100, 120, 50, 20, 40, 30 :: Int]) ] +dataframe> df = D.fromColumns ["product" =: [1, 1, 2, 2, 3, 3 :: Int], "amount" =: [100, 120, 50, 20, 40, 30 :: Int]] dataframe> df |> D.groupBy ["product"] |> ["total" .= F.countAll ] ``` ## Quick Start -Group sales by product and compute totals. The first block carries the -`scripths` cabal directives; each later section imports what it needs where it -first uses it. You can also drop the same code into an `Example.hs` and run it -with `cabal run Example.hs` after adding a `#!/usr/bin/env cabal` header. - ```haskell -- cabal: build-depends: dataframe, text -- cabal: default-extensions: OverloadedStrings, TypeApplications, TemplateHaskell, DataKinds, TypeFamilies, FlexibleInstances, FlexibleContexts, ScopedTypeVariables, DeriveGeneric, UndecidableInstances @@ -88,9 +97,11 @@ import qualified DataFrame.Functions as F import DataFrame.Expression.Operators -sales = D.fromNamedColumns - [ ("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int]) - , ("amount", D.fromList [100, 120, 50, 20, 40, 30 :: Int]) +import DataFrame ((=:)) + +sales = D.fromColumns + [ "product" =: [1, 1, 2, 2, 3, 3 :: Int] + , "amount" =: [100, 120, 50, 20, 40, 30 :: Int] ] -- Group by product and compute totals @@ -278,11 +289,6 @@ ### Generate a schema (and a row bridge) from a record ADT -When the canonical row shape lives in your code as a Haskell record, -`deriveSchemaFromType` produces both the typed schema and a `HasSchema` -instance that converts between `[Order]` and a `DataFrame` (or -`TypedDataFrame OrderSchema`) at runtime: - ```haskell import Data.Text (Text) import Data.Int (Int64) @@ -299,8 +305,8 @@ -- '[ '("order_id", Int64), '("region", Text), '("amount", Double)] -- instance DT.HasSchema Order where -- type Schema Order = OrderSchema --- toColumns = ... --- fromColumns = ... +-- recordsToColumns = ... +-- recordsFromFrame = ... xs :: [Order] xs = [Order 1 "us" 10.0, Order 2 "eu" 20.5] @@ -389,8 +395,8 @@ instance DT.HasSchema OrderG where type Schema OrderG = OrderGSchema - toColumns = DT.genericToColumns - fromColumns = DT.genericFromColumns + recordsToColumns = DT.genericToColumns + recordsFromFrame = DT.genericFromColumns ``` > @@ -442,9 +448,9 @@ ```haskell type ScoreSchema = '[ '("name", Text), '("score", Maybe Double)] -scoresDf = D.fromNamedColumns - [ ("name", D.fromList ["a", "b", "c" :: Text]) - , ("score", D.fromList [Just 1.0, Nothing, Just 3.0 :: Maybe Double]) +scoresDf = D.fromColumns + [ "name" =: ["a", "b", "c" :: Text] + , "score" =: [Just 1.0, Nothing, Just 3.0 :: Maybe Double] ] Just stdf = DT.freeze @ScoreSchema scoresDf @@ -459,26 +465,6 @@ > | -------------|-----------------|----------------- | > | a | 1.0 | 100.0 | > | c | 3.0 | 300.0 | - -## Features - -**I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from Hugging Face datasets (`hf://` URIs) via the `dataframe-huggingface` package. Column projection and predicate pushdown for Parquet reads. - -**Operations**: filter, select, derive, groupBy, aggregate, joins (inner, left, right, full outer), sort, sample, stratified sample, distinct, k-fold splits. - -**Expressions**: typed column references (`F.col @Double "x"`), arithmetic, comparisons, logical operators, nullable-aware three-valued logic (`.==`, `.&&`), string matching (`like`, `regex`), casting, and user-defined functions via `lift`/`lift2`. - -**Statistics**: mean, median, mode, variance, standard deviation, percentiles, inter-quartile range, correlation, skewness, frequency tables, imputation. - -**Plotting**: terminal plots (histogram, scatter, line, bar, box, pie, heatmap, stacked bar, correlation matrix) and interactive HTML plots. - -**Lazy engine**: streaming query execution for files that don't fit in memory. Rule-based optimizer with filter fusion, predicate pushdown, and dead column elimination. Pull-based executor with configurable batch sizes. - -**Interop**: Arrow C Data Interface for zero-copy round-trips with Python and Polars. - -**ML**: decision trees (TAO algorithm), feature synthesis, k-fold cross-validation, stratified sampling. - -**Notebooks**: IHaskell integration with [pre-built Binder examples](https://mybinder.org/v2/gh/mchav/ihaskell-dataframe/HEAD). ## Lazy Queries diff --git a/app/LazyBenchmark.hs b/app/LazyBenchmark.hs --- a/app/LazyBenchmark.hs +++ b/app/LazyBenchmark.hs @@ -2,15 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | End-to-end smoke test and benchmark for the Lazy streaming API: - generate a CSV then run five Lazy queries over it. - - Usage: - cabal run lazy-bench [-- [OPTIONS]] (+RTS -s -RTS for heap stats) - --rows N rows to generate (default 1_000_000_000) - --file PATH output CSV path (default: lazy_1b.csv in the system temp dir) - --skip-gen reuse the file if it already exists --} +-- | Benchmark for the Lazy API: generate a CSV, then run five Lazy queries. module Main where import Control.Monad (foldM, forM_, when) @@ -24,7 +16,7 @@ import DataFrame.Schema (Schema (..), schemaType) import System.Directory (doesFileExist, getFileSize, getTemporaryDirectory) import System.Environment (getArgs) -import System.Exit (exitFailure) +import System.Exit (exitFailure, exitSuccess) import System.IO ( BufferMode (..), IOMode (..), @@ -35,10 +27,6 @@ ) import System.Random.Stateful --- --------------------------------------------------------------------------- --- Defaults --- --------------------------------------------------------------------------- - defaultRows :: Int defaultRows = 1_000_000_000 @@ -49,9 +37,15 @@ chunkSize :: Int chunkSize = 500_000 --- --------------------------------------------------------------------------- --- Argument parsing --- --------------------------------------------------------------------------- +usage :: String +usage = + unlines + [ "Usage: cabal run lazy-bench [-- [OPTIONS]] (+RTS -s -RTS for heap stats)" + , " --rows N rows to generate (default 1_000_000_000)" + , " --file PATH output CSV path (default: lazy_1b.csv in the temp dir)" + , " --skip-gen reuse the file if it already exists" + , " --help show this message" + ] data Opts = Opts { optRows :: Int @@ -70,16 +64,7 @@ go opts ("--skip-gen" : rest) = go opts{optSkipGen = True} rest go _ (flag : _) = Left $ "unknown flag: " ++ flag --- --------------------------------------------------------------------------- --- CSV generation --- --------------------------------------------------------------------------- - -{- | Write @n@ rows of schema - - id (Int), x (Double), y (Double), category (Text: A/B/C/D) - - to @path@ using a streaming Builder, keeping heap usage constant. --} +-- | Stream @n@ rows of a four-column CSV to @path@ with constant heap. generateCsv :: FilePath -> Int -> IO () generateCsv path n = do g <- newIOGenM =<< newStdGen @@ -146,10 +131,6 @@ catChar 2 = Builder.char7 'C' catChar _ = Builder.char7 'D' --- --------------------------------------------------------------------------- --- Lazy queries --- --------------------------------------------------------------------------- - runQuery :: String -> IO D.DataFrame -> IO () runQuery label action = do putStrLn $ "\n── " ++ label @@ -162,17 +143,14 @@ putStrLn $ " time : " ++ showDiff t0 t1 when (rows > 0 && rows <= 30) $ print df --- --------------------------------------------------------------------------- --- Main --- --------------------------------------------------------------------------- - main :: IO () main = do hSetBuffering stdout LineBuffering args <- getArgs + when ("--help" `elem` args) $ putStr usage >> exitSuccess defFile <- defaultFile opts <- case parseArgs defFile args of - Left err -> putStrLn ("Error: " ++ err) >> exitFailure + Left err -> putStrLn ("Error: " ++ err) >> putStr usage >> exitFailure Right o -> return o let path = optFile opts @@ -244,10 +222,6 @@ $ L.scanCsv schema pathT putStrLn "\nDone." - --- --------------------------------------------------------------------------- --- Formatting helpers --- --------------------------------------------------------------------------- showDiff :: UTCTime -> UTCTime -> String showDiff t0 t1 = show (diffUTCTime t1 t0) diff --git a/app/Main.hs b/app/Main.hs --- a/app/Main.hs +++ b/app/Main.hs @@ -34,10 +34,8 @@ cabalFile = cacheDir "df-repl.cabal" mainFile = cacheDir "Main.hs" - -- Ensure the directory is a valid Cabal project with a local target to repl into. ensureBoilerplate projectFile cabalFile mainFile - -- Keep your ghci script updated (weekly). shouldDownload <- needsUpdate ghciScript when shouldDownload $ do putStrLn "\ESC[92mDownloading latest version of dataframe config...\ESC[0m" diff --git a/app/Synthesis.hs b/app/Synthesis.hs --- a/app/Synthesis.hs +++ b/app/Synthesis.hs @@ -26,7 +26,7 @@ "./data/titanic/test.csv" ) --- Survived is Maybe Int (safeRead = MaybeRead); prediction is Int (model output). +-- Survived is Maybe Int (safeRead = MaybeRead); prediction is Int. type RawPredSchema = '[ '("Survived", Maybe Int), '("prediction", Int)] @@ -124,9 +124,7 @@ (w : _) -> T.dropEnd 1 w [] -> "" -{- | Compute binary classification accuracy from a DataFrame containing - "Survived" and "prediction" columns. --} +-- | Binary classification accuracy from "Survived" and "prediction" columns. computeAccuracy :: D.DataFrame -> Double computeAccuracy df = let tdf = diff --git a/benchmark/Main.hs b/benchmark/Main.hs --- a/benchmark/Main.hs +++ b/benchmark/Main.hs @@ -9,17 +9,13 @@ import Control.DeepSeq (NFData (..)) import Control.Monad (void) import Criterion.Main +import DataFrame ((=:)) import DataFrame.Expression.Operators import DataFrame.Internal.DataFrame (forceDataFrame) import DataFrame.Operations.Join import System.Process hiding (env) import System.Random.Stateful -{- | Criterion's 'nf' and 'env' force benchmark inputs/results to normal form -via 'NFData'. The core library intentionally dropped its @instance NFData -DataFrame@ in favour of 'forceDataFrame', so we provide a thin orphan here -(scoped to the benchmark) that reuses it. --} instance NFData D.DataFrame where rnf df = forceDataFrame df `seq` () @@ -107,10 +103,7 @@ parseMeasurementsTXT :: IO () parseMeasurementsTXT = parseFile "./data/measurements.txt" -{- | Generate a pair of dataframes for a 1:1 join scenario. - Left has keys [0..n-1], right has keys [0..n-1] shuffled. - overlap controls what fraction of keys appear in both sides. --} +-- | Two dataframes for a 1:1 join; overlap is the share of shared keys. mkOneToOne :: Int -> Double -> IO (D.DataFrame, D.DataFrame) mkOneToOne n overlap = do g <- newIOGenM =<< newStdGen @@ -119,23 +112,21 @@ let leftKeys = [0 :: Int .. n - 1] leftVals = [0 :: Int .. n - 1] leftDf = - D.fromNamedColumns - [ ("key", D.fromList leftKeys) - , ("A", D.fromList leftVals) + D.fromColumns + [ "key" =: leftKeys + , "A" =: leftVals ] -- Right: take first `rightSize` keys, add non-overlapping keys for the rest rightPayload <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightSize] let rightKeys = [0 :: Int .. rightSize - 1] rightDf = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList rightKeys) , ("B", D.fromList rightPayload) ] return (leftDf, rightDf) -{- | Generate a pair of dataframes for a many-to-many join scenario. - Keys are drawn from [0..cardinality-1], so rows share keys. --} +-- | Two dataframes for a many-to-many join with keys in [0..cardinality-1]. mkManyToMany :: Int -> Int -> Int -> IO (D.DataFrame, D.DataFrame) mkManyToMany leftRows rightRows cardinality = do g <- newIOGenM =<< newStdGen @@ -145,22 +136,18 @@ leftVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. leftRows] rightVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightRows] let leftDf = - D.fromNamedColumns - [ ("key", D.fromList leftKeys) - , ("A", D.fromList leftVals) + D.fromColumns + [ "key" =: leftKeys + , "A" =: leftVals ] rightDf = - D.fromNamedColumns - [ ("key", D.fromList rightKeys) - , ("B", D.fromList rightVals) + D.fromColumns + [ "key" =: rightKeys + , "B" =: rightVals ] return (leftDf, rightDf) -{- | Generate a pair of dataframes for a many-to-one join - (fact table joining a dimension table). - Left has n rows with keys drawn from [0..dimSize-1]. - Right has exactly dimSize rows with unique keys. --} +-- | Fact table and dimension table (unique keys) for a many-to-one join. mkManyToOne :: Int -> Int -> IO (D.DataFrame, D.DataFrame) mkManyToOne factRows dimSize = do g <- newIOGenM =<< newStdGen @@ -168,14 +155,14 @@ factVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. factRows] dimVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. dimSize] let factDf = - D.fromNamedColumns - [ ("key", D.fromList factKeys) - , ("A", D.fromList factVals) + D.fromColumns + [ "key" =: factKeys + , "A" =: factVals ] dimDf = - D.fromNamedColumns - [ ("key", D.fromList [0 :: Int .. dimSize - 1]) - , ("B", D.fromList dimVals) + D.fromColumns + [ "key" =: [0 :: Int .. dimSize - 1] + , "B" =: dimVals ] return (factDf, dimDf) @@ -189,16 +176,16 @@ lv <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. leftRows] rv <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightRows] let leftDf = - D.fromNamedColumns - [ ("key1", D.fromList lk1) - , ("key2", D.fromList lk2) - , ("A", D.fromList lv) + D.fromColumns + [ "key1" =: lk1 + , "key2" =: lk2 + , "A" =: lv ] rightDf = - D.fromNamedColumns - [ ("key1", D.fromList rk1) - , ("key2", D.fromList rk2) - , ("B", D.fromList rv) + D.fromColumns + [ "key1" =: rk1 + , "key2" =: rk2 + , "B" =: rv ] return (leftDf, rightDf) diff --git a/dataframe.cabal b/dataframe.cabal --- a/dataframe.cabal +++ b/dataframe.cabal @@ -1,6 +1,6 @@ cabal-version: 3.4 name: dataframe -version: 3.6.1.0 +version: 3.7.0.0 synopsis: A fast, safe, and intuitive DataFrame library. description: A fast, safe, and intuitive DataFrame library for exploratory data analysis. @@ -127,13 +127,13 @@ DataFrame.Typed.Record, DataFrame.Typed.Generic build-depends: base >= 4 && <5, - dataframe-core >= 2.5 && < 2.6, + dataframe-core >= 2.6 && < 2.7, dataframe-json >= 1.2.0.1 && < 1.3, - dataframe-expr-serializer >= 1.2.1 && < 1.3, - dataframe-operations >= 2.5 && < 2.6, + dataframe-expr-serializer >= 1.3 && < 1.4, + dataframe-operations >= 2.6 && < 2.7, dataframe-parsing >= 2.2 && < 2.3, dataframe-viz >= 1.3.2 && < 1.4, - dataframe-learn >= 2.4.2 && < 2.5 + dataframe-learn >= 2.5 && < 2.6 if !flag(no-csv) reexported-modules: DataFrame.IO.CSV, @@ -157,7 +157,7 @@ DataFrame.IO.Parquet.Time, DataFrame.IO.Utils.RandomAccess, DataFrame.Typed.IO.Parquet - build-depends: dataframe-parquet >= 1.5.0.1 && < 1.6 + build-depends: dataframe-parquet >= 1.6 && < 1.7 cpp-options: -DWITH_PARQUET -- The lazy executor calls both CSV and Parquet readers directly, so @@ -191,8 +191,8 @@ import: warnings main-is: Benchmark.hs build-depends: base >= 4 && < 5, - dataframe >= 3.6 && < 3.7, - dataframe-operations >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, time >= 1.12 && < 2, vector >= 0.13 && < 0.15, @@ -204,10 +204,10 @@ import: warnings main-is: Synthesis.hs build-depends: base >= 4 && < 5, - dataframe >= 3.6 && < 3.7, - dataframe-core >= 2.5 && < 2.6, - dataframe-learn >= 2.4.2 && < 2.5, - dataframe-operations >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-core >= 2.6 && < 2.7, + dataframe-learn >= 2.5 && < 2.6, + dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, text >= 2.1 && < 3 hs-source-dirs: app @@ -238,8 +238,8 @@ build-depends: base >= 4 && < 5, bytestring >= 0.11 && < 0.14, containers >= 0.6.7 && < 0.10, - dataframe >= 3.6 && < 3.7, - dataframe-core >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-core >= 2.6 && < 2.7, dataframe-lazy >= 2.4.1 && < 2.5, dataframe-parsing >= 2.2 && < 2.3, directory >= 1.3.0.0 && < 2, @@ -259,9 +259,9 @@ criterion >= 1 && < 2, deepseq >= 1.4 && < 2, process >= 1.6 && < 2, - dataframe >= 3.6 && < 3.7, - dataframe-core >= 2.5 && < 2.6, - dataframe-operations >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-core >= 2.6 && < 2.7, + dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, default-language: Haskell2010 ghc-options: @@ -298,12 +298,17 @@ Learn.SklearnParity, Learn.Synthesis, Learn.MetricsTests, + Operations.ExprParse, Learn.Metamorphic, IO.CSV, IO.CsvGolden, IO.JSON, IR.ExprJsonRoundtrip, Operations.Aggregations, + Operations.CountDistinct, + Operations.SortMask, + Operations.TextKernels, + Operations.TopBy, Operations.Apply, Operations.Core, Operations.Derive, @@ -329,6 +334,9 @@ Operations.Take, Operations.Typing, Operations.VectorKernel, + Operations.InterpreterKernels, + Operations.MonomorphicLoops, + Operations.RowWise, Operations.Record, LazyParquet, LazyParity, @@ -345,17 +353,17 @@ build-depends: base >= 4 && < 5, aeson >= 0.11.0.0 && < 3, bytestring >= 0.11 && < 0.14, - dataframe >= 3.6 && < 3.7, - dataframe-core >= 2.5 && < 2.6, - dataframe-core >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-core >= 2.6 && < 2.7, + dataframe-core >= 2.6 && < 2.7, dataframe-csv >= 2.3 && < 2.4, - dataframe-expr-serializer >= 1.2.1 && < 1.3, + dataframe-expr-serializer >= 1.3 && < 1.4, dataframe-fastcsv >= 1.4.0.1 && < 1.5, dataframe-json >= 1.2.0.1 && < 1.3, dataframe-lazy >= 2.4.1 && < 2.5, - dataframe-learn >= 2.4.2 && < 2.5, - dataframe-operations >= 2.5 && < 2.6, - dataframe-parquet >= 1.5.0.1 && < 1.6, + dataframe-learn >= 2.5 && < 2.6, + dataframe-operations >= 2.6 && < 2.7, + dataframe-parquet >= 1.6 && < 1.7, dataframe-parsing >= 2.2 && < 2.3, HUnit >= 1.6 && < 1.8, QuickCheck >= 2 && < 3, @@ -381,9 +389,9 @@ other-modules: Internal.PackedText build-depends: base >= 4 && < 5, bytestring >= 0.11 && < 0.14, - dataframe >= 3.6 && < 3.7, - dataframe-core >= 2.5 && < 2.6, - dataframe-operations >= 2.5 && < 2.6, + dataframe >= 3.7 && < 3.8, + dataframe-core >= 2.6 && < 2.7, + dataframe-operations >= 2.6 && < 2.7, HUnit >= 1.6 && < 1.8, text >= 2.1 && < 3, vector >= 0.13 && < 0.15 diff --git a/src/DataFrame.hs b/src/DataFrame.hs --- a/src/DataFrame.hs +++ b/src/DataFrame.hs @@ -1,17 +1,6 @@ {-# LANGUAGE CPP #-} -{- | -Module : DataFrame -Copyright : (c) 2024 - 2026 Michael Chavinda -License : GPL-3.0 -Maintainer : mschavinda@gmail.com -Stability : experimental -Portability : POSIX - -Batteries-included entry point for @dataframe@: re-exports the most commonly -used pieces for GHCi and scripts. Use the @D.@ prefix for core table operations -and @F.@ for the expression DSL. --} +-- | Batteries-included entry point re-exporting the common pieces for GHCi. module DataFrame ( -- * Core data structures module CoreTypes, @@ -79,9 +68,7 @@ ) where --- DecisionTree's own `percentile`/`percentiles` are hidden here; the public --- versions come from Operations.Statistics / DataFrame.Synthesis, avoiding --- ambiguous-export errors. +-- DecisionTree's `percentile`/`percentiles` are hidden to avoid ambiguity. import DataFrame.DecisionTree as DecisionTree import DataFrame.Display as Display ( DisplayOptions (..), @@ -111,8 +98,7 @@ readJSON, readJSONEither, ) --- Marks the dataframe-expr-serializer dependency as used (the codec modules are --- surfaced to consumers via this library's @reexported-modules@). +-- Marks the dataframe-expr-serializer dependency as used. import DataFrame.Expr.Serialize () #ifdef WITH_PARQUET import DataFrame.IO.Parquet as Parquet ( @@ -144,13 +130,13 @@ columnNames, defaultTruncateConfig, toNamedExpr, - toUExpr, - fromUExpr, + toSomeExpr, + fromSomeExpr, eSize, empty, fromAny, fromList, - fromNamedColumns, + fromColumns, fromUnboxedVector, fromVector, hasElemType, @@ -172,6 +158,7 @@ toRowVector, toSeparated, toVector, + (=:), ) import DataFrame.Schema as Schema ( SchemaType (..), @@ -182,9 +169,7 @@ import DataFrame.Typed.TH.Records as SchemaTH (deriveSchemaFromType, deriveSchemaValues) #endif #ifdef WITH_LAZY --- Re-export only the lazy engine's types and source constructors; its --- operator surface (filter/select/derive/...) collides with the eager API, --- so users wanting full lazy access `import qualified DataFrame.Lazy as L`. +-- Re-export only the lazy types and sources; its operators clash with eager. import DataFrame.Lazy as Lazy ( LazyDataFrame, fromDataFrame, @@ -221,6 +206,7 @@ SortOrder (..), shuffle, sortBy, + topBy, ) import DataFrame.Operations.Statistics as Statistics ( correlation, diff --git a/src/DataFrame/TH.hs b/src/DataFrame/TH.hs --- a/src/DataFrame/TH.hs +++ b/src/DataFrame/TH.hs @@ -1,21 +1,6 @@ {-# LANGUAGE CPP #-} -{- | -Module : DataFrame.TH -License : MIT - -Backwards-compatibility re-export hub for the split @DataFrame.TH.*@ -modules. New code can import the specific submodule directly: - - * "DataFrame.TH.Records" — record/DataFrame-based splices (no file IO) - * "DataFrame.TH.CSV" — CSV-file-based splices (requires @-fwith-csv@) - * "DataFrame.TH.Parquet" — Parquet-file-based splices (requires @-fwith-parquet@) - -These live in @dataframe-th@, @dataframe-csv-th@, and -@dataframe-parquet-th@ respectively. The CSV/Parquet re-exports are -guarded with the @CSV_TH@ / @PARQUET_TH@ CPP defines that the -meta-@dataframe@ package sets based on its cabal flags. --} +-- | Backwards-compatibility re-export hub for the split TH modules. module DataFrame.TH ( module DataFrame.TH.Records, #ifdef WITH_CSV_TH diff --git a/src/DataFrame/Typed.hs b/src/DataFrame/Typed.hs --- a/src/DataFrame/Typed.hs +++ b/src/DataFrame/Typed.hs @@ -1,62 +1,7 @@ {-# LANGUAGE CPP #-} {-# LANGUAGE DataKinds #-} -{- | -Module : DataFrame.Typed -Copyright : (c) 2024 - 2026 Michael Chavinda -License : MIT -Maintainer : mschavinda@gmail.com -Stability : experimental - -A type-safe layer over the @dataframe@ library. - -This module provides 'TypedDataFrame', a phantom-typed wrapper around -the untyped 'DataFrame' that tracks column names and types at compile time. -All operations delegate to the untyped core at runtime; the phantom type -is updated at compile time to reflect schema changes. - -== Key difference from untyped API: TExpr - -All expression-taking operations use 'TExpr' (typed expressions) instead -of raw @Expr@. Column references are validated at compile time: - -@ -{\-\# LANGUAGE DataKinds, TypeApplications, TypeOperators \#-\} -import qualified DataFrame.Typed as T - -type People = '[ '(\"name\", Text), '(\"age\", Int)] - -main = do - raw <- D.readCsv \"people.csv\" - case T.freeze \@People raw of - Nothing -> putStrLn \"Schema mismatch!\" - Just df -> do - let adults = T.filterWhere (T.col \@\"age\" T..>=. T.lit 18) df - let names = T.columnAsList \@\"name\" adults -- :: [Text] - print names -@ - -Column references like @T.col \@\"age\"@ are checked at compile time — if the -column doesn't exist or has the wrong type, you get a type error, not a -runtime exception. - -== filterAllJust tracks Maybe-stripping - -@ -df :: TypedDataFrame '[ '(\"x\", Maybe Double), '(\"y\", Int)] -T.filterAllJust df :: TypedDataFrame '[ '(\"x\", Double), '(\"y\", Int)] -@ - -== Typed aggregation - -@ -result = T.aggregate - ( T.as \@\"total\" (T.sum (T.col \@\"salary\")) - . T.as \@\"count\" (T.count (T.col \@\"salary\")) - ) - (T.groupBy \@'[\"dept\"] employees) -@ --} +-- | A type-safe layer that tracks column names and types over the core API. module DataFrame.Typed ( -- * Core types TypedDataFrame, @@ -68,6 +13,8 @@ col, lit, ifThenElse, + caseWhen, + orElse, lift, lift2, nullLift, diff --git a/src/DataFrame/Typed/TH.hs b/src/DataFrame/Typed/TH.hs --- a/src/DataFrame/Typed/TH.hs +++ b/src/DataFrame/Typed/TH.hs @@ -1,19 +1,6 @@ {-# LANGUAGE CPP #-} -{- | -Module : DataFrame.Typed.TH -License : MIT - -Backwards-compatibility re-export hub for the split @DataFrame.Typed.TH.*@ -modules: - - * "DataFrame.Typed.TH.Records" — record-based schema derivation (no file IO) - * "DataFrame.Typed.TH.CSV" — CSV-file-based schema derivation (requires @-fwith-csv@) - -These live in @dataframe-th@ and @dataframe-csv-th@ respectively. The CSV -re-export is guarded with the @CSV_TH@ CPP define that the -meta-@dataframe@ package sets based on its cabal flags. --} +-- | Backwards-compatibility re-export hub for @DataFrame.Typed.TH.*@ modules. module DataFrame.Typed.TH ( module DataFrame.Typed.TH.Records, #ifdef WITH_CSV_TH diff --git a/tests/Assertions.hs b/tests/Assertions.hs --- a/tests/Assertions.hs +++ b/tests/Assertions.hs @@ -5,9 +5,10 @@ import qualified Data.List as L import Control.Exception +import GHC.Float (castDoubleToWord64, castFloatToWord32) import Test.HUnit --- Adapted from: https://github.com/BartMassey/chunk/blob/1ee4bd6545e0db6b8b5f4935d97e7606708eacc9/hunit.hs#L29 +-- Adapted from BartMassey/chunk, hunit.hs. assertExpectException :: String -> String -> @@ -33,3 +34,37 @@ ++ expected ++ ", got: " ++ show e + +-- | Equality that tells NaNs and signed zeros apart for floating types. +class BitEq a where + bitEq :: a -> a -> Bool + +instance BitEq Double where + bitEq a b = castDoubleToWord64 a == castDoubleToWord64 b +instance BitEq Float where + bitEq a b = castFloatToWord32 a == castFloatToWord32 b +instance BitEq Int where + bitEq = (==) +instance BitEq Bool where + bitEq = (==) +instance (BitEq a) => BitEq (Maybe a) where + bitEq (Just a) (Just b) = bitEq a b + bitEq Nothing Nothing = True + bitEq _ _ = False + +-- | Lists equal under 'bitEq'; on failure, reports the first differing row. +assertBitEq :: (BitEq a, Show a) => String -> [a] -> [a] -> Assertion +assertBitEq label expected actual + | length expected /= length actual = + assertFailure + ( label + ++ ": " + ++ show (length actual) + ++ " rows, expected " + ++ show (length expected) + ) + | otherwise = case [(i, e, a) | (i, e, a) <- zip3 [0 :: Int ..] expected actual, not (bitEq e a)] of + [] -> pure () + (i, e, a) : _ -> + assertFailure + (label ++ ": row " ++ show i ++ " expected " ++ show e ++ ", got " ++ show a) diff --git a/tests/Functions.hs b/tests/Functions.hs --- a/tests/Functions.hs +++ b/tests/Functions.hs @@ -3,6 +3,7 @@ module Functions where +import Data.Time.Calendar (Day, fromGregorian) import qualified DataFrame as D import DataFrame.Functions ( sanitize, @@ -63,7 +64,7 @@ ] df :: D.DataFrame df = - D.fromNamedColumns + D.fromColumns [("A", DI.fromList [(1 :: Int) .. 10])] testSum :: Test @@ -71,7 +72,7 @@ TestCase ( assertEqual "Sum first 10 numbers" - ( D.fromNamedColumns + ( D.fromColumns [ ("A", DI.fromList [(1 :: Int) .. 10]) , ("sum", DI.fromList (replicate 10 (55 :: Int))) ] @@ -79,8 +80,40 @@ (D.derive "sum" (F.sum (F.col @Int "A")) df) ) +testPow :: Test +testPow = + TestCase + ( assertEqual + "pow of a compound base" + [4, 9, 16, 25, 36, 49, 64, 81, 100, 121] + ( D.columnAsList @Int + (F.col "sq") + (D.derive "sq" (F.pow (F.col @Int "A" + F.lit 1) 2) df) + ) + ) + +testDaysBetween :: Test +testDaysBetween = + TestCase + ( assertEqual + "daysBetween d1 d2 is d1 minus d2 in either argument order" + ([-9], [9]) + ( days (F.col @Day "start") (F.col @Day "end") + , days (F.col @Day "end") (F.col @Day "start") + ) + ) + where + dates = + D.fromColumns + [ ("start", DI.fromList [fromGregorian 2024 3 1]) + , ("end", DI.fromList [fromGregorian 2024 3 10]) + ] + days a b = D.columnAsList @Int (F.col "d") (D.derive "d" (F.daysBetween a b) dates) + tests :: [Test] tests = [ TestLabel "sanitizeIdentifiers" sanitizeIdentifiers , TestLabel "testSum" testSum + , TestLabel "testPow" testPow + , TestLabel "testDaysBetween" testDaysBetween ] diff --git a/tests/IO/CSV.hs b/tests/IO/CSV.hs --- a/tests/IO/CSV.hs +++ b/tests/IO/CSV.hs @@ -46,7 +46,7 @@ fromCsvRoundTrip :: Test fromCsvRoundTrip = TestLabel "fromCsv_roundTrip" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList @Int [1, 2, 3]) , ("b", DI.fromList @T.Text ["hello", "world", "test"]) ] @@ -65,7 +65,7 @@ fromCsvBytesRoundTrip :: Test fromCsvBytesRoundTrip = TestLabel "fromCsvBytes_roundTrip" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList @Int [10, 20]) , ("y", DI.fromList @Double [1.5, 2.5]) ] diff --git a/tests/IO/CsvGolden.hs b/tests/IO/CsvGolden.hs --- a/tests/IO/CsvGolden.hs +++ b/tests/IO/CsvGolden.hs @@ -2,10 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | Golden semantics for the default CSV reader, pinned against the -cassava-based oracle (2026-06-12) before the strict-scanner rewrite. -Ragged-row cases encode the one intentional change: pad-with-null (audit D6). --} +-- | Golden semantics for the default CSV reader, pinned against the old oracle. module IO.CsvGolden (tests) where import qualified Data.ByteString as BS @@ -586,8 +583,7 @@ assertEqual "dims" (2, 2) (dataframeDimensions df) assertEqual "index map" [("a", 1)] (M.toList (columnIndices df)) --- Ingest pin: a string column freezes to a shared-buffer 'PackedText' --- (no per-row boxed Text header), while values stay byte-identical. +-- Ingest pin: string columns freeze to a shared-buffer 'PackedText'. textFreezesPacked :: Test textFreezesPacked = TestLabel "text_freezes_packed" $ TestCase $ do df <- decodeSeparated defaultReadOptions (BL.fromStrict "a,b\n1,x\n2,y\n") @@ -595,7 +591,7 @@ Nothing -> assertFailure "missing text column" Just col -> do assertBool "text column is PackedText" (DI.isPackedText col) - assertEqual "values byte-identical" (texts ["x", "y"]) col + assertEqual "values equal" (texts ["x", "y"]) col tests :: [Test] tests = dupHeaders : textFreezesPacked : map goldenCase goldenCases diff --git a/tests/IO/JSON.hs b/tests/IO/JSON.hs --- a/tests/IO/JSON.hs +++ b/tests/IO/JSON.hs @@ -46,13 +46,13 @@ Right df -> do assertEqual "Missing key: 2 rows" 2 (D.nRows df) assertEqual "Missing key: 2 columns" 2 (D.nColumns df) - -- 'b' is absent from the second row, so must have a missing value + -- 'b' is absent from the second row, so it is missing assertBool "b column should have missing values" (maybe False DI.hasMissing (DI.getColumn "b" df)) ) --- | When column values are different types across rows, the column becomes generic. +-- | Column values of different types across rows make the column generic. jsonMixedTypeColumn :: Test jsonMixedTypeColumn = TestCase diff --git a/tests/IR/ExprJsonRoundtrip.hs b/tests/IR/ExprJsonRoundtrip.hs --- a/tests/IR/ExprJsonRoundtrip.hs +++ b/tests/IR/ExprJsonRoundtrip.hs @@ -3,9 +3,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Round-trip, pipeline, end-to-end, and wire-shape contract tests for the -expression/pipeline serializer. --} +-- | Round-trip and wire-shape contract tests for the expression serializer. module IR.ExprJsonRoundtrip (tests) where import qualified Data.Aeson as Aeson @@ -19,7 +17,7 @@ import qualified DataFrame.Functions as F import DataFrame.Internal.Column (Columnable, TypedColumn (..), toVector) import qualified DataFrame.Internal.Column as DI -import DataFrame.Internal.Expression (Expr, UExpr (..)) +import DataFrame.Internal.Expression (Expr, SomeExpr (..)) import DataFrame.Internal.Interpreter (interpret) import DataFrame.LinearModel (defaultLinearConfig) @@ -31,7 +29,6 @@ ) import DataFrame.Expr.Serialize ( - SomeExpr (..), decodeExprAny, decodeExprAt, decodeExprFromBytes, @@ -52,11 +49,11 @@ close a b = length a == length b && and (zipWith (\x y -> abs (x - y) <= 1e-9) a b) df3 :: D.DataFrame -df3 = D.fromNamedColumns [("x", DI.fromList [1, 2, 3 :: Double])] +df3 = D.fromColumns [("x", DI.fromList [1, 2, 3 :: Double])] regDF :: D.DataFrame regDF = - D.fromNamedColumns + D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2]) @@ -65,13 +62,11 @@ xs1 = [1, 2, 3, 4, 5, 6, 7, 8] :: [Double] xs2 = [2, 1, 4, 3, 6, 5, 8, 7] :: [Double] -{- | An expression round-trips when encode → decode → re-encode yields the same -JSON. Avoids needing an 'Eq' instance on the GADT. --} +-- | Round-trips when encode, decode, re-encode yields the same JSON. roundtrips :: (Columnable a) => Expr a -> Bool roundtrips e = fromRight False $ do v1 <- encodeExpr e - SomeExpr _ e' <- decodeExprAny v1 + SomeExpr e' <- decodeExprAny v1 v2 <- encodeExpr e' Right (v1 == v2) @@ -109,8 +104,7 @@ , testCase "over" (roundtrips (F.over ["g"] (F.sum (F.col @Double "x")))) ] --- Documented limitation: collect's list-typed output has no type tag, so it --- cannot be encoded. +-- Documented limitation: collect's list-typed output cannot be encoded. limitationTests :: [Test] limitationTests = [ testCase @@ -161,8 +155,8 @@ , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do let fp = dir ++ "/pipeline.json" nes = - [ ("z", UExpr (F.col @Double "x" * F.lit 2)) - , ("w", UExpr (F.col @Double "x" + F.lit 10)) + [ ("z", SomeExpr (F.col @Double "x" * F.lit 2)) + , ("w", SomeExpr (F.col @Double "x" + F.lit 10)) ] sr <- savePipelineToFile fp nes assertEqual "save pipeline ok" (Right ()) sr @@ -175,7 +169,7 @@ assertBool "w = x+10" (close (interpD df' (F.col @Double "w")) [11, 12, 13]) , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do let fp = dir ++ "/transform.json" - t = Transform [("z", UExpr (F.col @Double "x" * F.lit 2))] + t = Transform [("z", SomeExpr (F.col @Double "x" * F.lit 2))] sr <- saveTransformToFile fp t assertEqual "save transform ok" (Right ()) sr loaded <- loadTransformFromFile fp diff --git a/tests/Internal/ColumnBuilder.hs b/tests/Internal/ColumnBuilder.hs --- a/tests/Internal/ColumnBuilder.hs +++ b/tests/Internal/ColumnBuilder.hs @@ -100,7 +100,6 @@ | otherwise = fromVector (VB.fromList (catMaybes xs)) -- Read a text column back as a Maybe list, respecting the bitmap. --- Ingest now freezes text to 'PackedText'; materialize to the boxed form. columnTexts :: Column -> [Maybe T.Text] columnTexts c@(PackedText _ _) = columnTexts (materializePacked c) columnTexts (BoxedColumn mb (v :: VB.Vector b)) = @@ -191,8 +190,7 @@ (map Just ts) (columnTexts (buildTextColumn (map Just ts))) --- A multibyte sequence split across two fields: the whole buffer is valid --- UTF-8 but each field alone is not. Both must lenient-decode to U+FFFD. +-- A multibyte sequence split across fields must lenient-decode to U+FFFD. splitMultibyteAcrossFields :: Test splitMultibyteAcrossFields = TestCase $ do let fields = [[0xC2], [0x80]] diff --git a/tests/Internal/DictEncode.hs b/tests/Internal/DictEncode.hs --- a/tests/Internal/DictEncode.hs +++ b/tests/Internal/DictEncode.hs @@ -1,13 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ScopedTypeVariables #-} -{- | Correctness oracle for the dictionary-encode building block -('DataFrame.Internal.Column.Encode'). A text column encodes to dense -first-appearance @Int@ codes: two rows share a code iff their text is equal, the -codes are a contiguous @0 .. card-1@ range in first-appearance order, packed and -boxed Text encode identically, and the cap parameter bails past a cardinality. -Pins the step that the group-by planner can route a text key through. --} +-- | Correctness oracle for dictionary-encoding text columns into dense codes. module Internal.DictEncode (tests) where import qualified Data.ByteString as B @@ -56,9 +50,7 @@ boxedFromTexts :: [T.Text] -> DI.Column boxedFromTexts = DI.fromList -{- | The reference dense first-appearance encoding computed with a plain -'Data.Map': scan in row order, assign the next id to each new value. --} +-- | The reference first-appearance encoding, computed with a plain Map. oracleCodes :: [T.Text] -> ([Int], Int) oracleCodes ts = go ts M.empty 0 [] where diff --git a/tests/Internal/Markdown.hs b/tests/Internal/Markdown.hs --- a/tests/Internal/Markdown.hs +++ b/tests/Internal/Markdown.hs @@ -1,8 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | Markdown rendering must escape pipe-table metacharacters in cell values, -or an operator name like @<|>@ splits the row into the wrong columns. --} +-- | Markdown rendering must escape pipe-table metacharacters in cell values. module Internal.Markdown (tests) where import qualified Data.Text as T @@ -12,8 +10,7 @@ import qualified DataFrame.Internal.DataFrame as D import Test.HUnit --- Count the column-delimiter pipes in a rendered row: a bare '|' delimits a --- cell; an escaped "\|" does not. +-- Count a rendered row's delimiter pipes; an escaped "\|" is not one. unescapedPipes :: T.Text -> Int unescapedPipes = go . T.unpack where @@ -31,7 +28,7 @@ , TestLabel "toMarkdown keeps an operator name in one cell" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("name", fromList ["plain" :: T.Text, "f<|>g"]) , ("mb", fromList [1.0 :: Double, 2.0]) ] @@ -45,7 +42,7 @@ , TestLabel "nullable boxed Text renders without quotes" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [("name", ensureOptional (fromList ["Ada" :: T.Text]))] md = D.toMarkdown df assertBool diff --git a/tests/Internal/PackedText.hs b/tests/Internal/PackedText.hs --- a/tests/Internal/PackedText.hs +++ b/tests/Internal/PackedText.hs @@ -2,12 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | Correctness oracle for the packed-text column variant. Every assertion -builds a 'PackedText' column from raw bytes + offsets and the same data as a -'BoxedColumn Text', then checks display, extraction, equality, ordering, -hashing, groupBy and join all agree. This pins behavioral parity before any -reader emits packed columns. --} +-- | Correctness oracle: packed-text columns behave like boxed Text columns. module Internal.PackedText (tests) where import qualified Data.Text as T @@ -20,6 +15,7 @@ import Data.Text.Encoding (encodeUtf8) import Data.Word (Word8) import qualified DataFrame as D +import DataFrame.Expression.Operators ((./=.), (.==.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI import DataFrame.Internal.Data.PackedText (mkPackedContiguous) @@ -67,8 +63,8 @@ -- Single-column dataframes for groupBy / hashing. packedDf, boxedDf :: D.DataFrame -packedDf = D.fromNamedColumns [("k", packedCol)] -boxedDf = D.fromNamedColumns [("k", boxedCol)] +packedDf = D.fromColumns [("k", packedCol)] +boxedDf = D.fromColumns [("k", boxedCol)] displayParity :: Test displayParity = @@ -116,15 +112,14 @@ joinParity :: Test joinParity = TestCase $ do - let lhsP = D.fromNamedColumns [("k", packedCol)] - lhsB = D.fromNamedColumns [("k", boxedCol)] - rhs = D.fromNamedColumns [("k", boxedFromTexts ["apple", "banana", "日本語"])] + let lhsP = D.fromColumns [("k", packedCol)] + lhsB = D.fromColumns [("k", boxedCol)] + rhs = D.fromColumns [("k", boxedFromTexts ["apple", "banana", "日本語"])] joinedP = D.innerJoin ["k"] lhsP rhs joinedB = D.innerJoin ["k"] lhsB rhs assertBool "inner join parity" (joinedP == joinedB) --- A gather (atIndicesStable) on a packed column stays packed and equals the --- boxed gather. Indices repeat, reorder, and drop rows. +-- A gather on a packed column stays packed and equals the boxed gather. gatherPreservesPacked :: Test gatherPreservesPacked = TestCase $ do let ixs = VU.fromList [9, 0, 0, 5, 7, 2, 3 :: Int] @@ -147,19 +142,18 @@ (DI.isPackedText (unsafeGetColumn "k" sortedP)) assertBool "sorted packed == sorted boxed" (sortedP == sortedB) --- A left join (sentinel gather, -1 for missing right rows) on a packed right --- column stays packed, builds the correct null bitmap, and equals boxed. +-- A left join on a packed right column stays packed and equals boxed. leftJoinSentinelPreservesPacked :: Test leftJoinSentinelPreservesPacked = TestCase $ do - let lhsP = D.fromNamedColumns [("k", packedCol)] - lhsB = D.fromNamedColumns [("k", boxedCol)] + let lhsP = D.fromColumns [("k", packedCol)] + lhsB = D.fromColumns [("k", boxedCol)] rhsP = - D.fromNamedColumns + D.fromColumns [ ("k", packedFromTexts ["apple", "banana", "日本語"]) , ("v", packedFromTexts ["RA", "RB", "RC"]) ] rhsB = - D.fromNamedColumns + D.fromColumns [ ("k", boxedFromTexts ["apple", "banana", "日本語"]) , ("v", boxedFromTexts ["RA", "RB", "RC"]) ] @@ -170,8 +164,7 @@ (DI.isPackedText (unsafeGetColumn "v" joinedP)) assertBool "left join packed == boxed" (joinedP == joinedB) --- A slice of a packed column stays packed: decoding to Text would make a --- ten-row slice cost the whole frame. +-- A slice of a packed column stays packed instead of decoding to Text. slicePreservesPacked :: Test slicePreservesPacked = TestCase $ do let sp = DI.sliceColumn 3 4 packedCol @@ -194,8 +187,7 @@ (DI.toList @T.Text tb) (DI.toList @T.Text tp) --- Every other representation rejects a slice running past the end; the packed --- arm must not quietly decode the overrun as empty strings. +-- A packed slice past the end must be rejected, not decoded as empty. sliceRejectsInvalidBounds :: Test sliceRejectsInvalidBounds = TestCase $ do r <- try (evaluate (DI.columnLength (DI.sliceColumn 8 5 packedCol))) @@ -204,6 +196,48 @@ Right len -> assertFailure ("expected an invalid slice, got length " ++ show len) +-- Rows for the eq/neq kernel: valid text, empty, and invalid UTF-8. +kernelRowBytes :: [[Word8]] +kernelRowBytes = + map (B.unpack . encodeUtf8) ["apple", "", "café", "日本語", "apple", "\xFFFD"] + ++ [[0xFF], [0x63, 0x61, 0x66, 0xC3], [0x80], [0xEF, 0xBF]] + +packedFromBytes :: [[Word8]] -> DI.Column +packedFromBytes rows = + DI.PackedText + Nothing + ( mkPackedContiguous + (arrayFromBytes (concat rows)) + (VU.fromList (scanl (+) 0 (map length rows))) + ) + +-- The eq/neq-literal kernel must agree with the decoded-column expression. +eqLiteralKernelParity :: Test +eqLiteralKernelParity = TestCase $ do + let base = D.fromColumns [("k", packedFromBytes kernelRowBytes)] + sorted = D.sortBy [D.Asc (F.col @T.Text "k")] base + boxed df = D.fromColumns [("k", DI.materializePacked (unsafeGetColumn "k" df))] + lits = ["apple", "", "café", "日本語", "\xFFFD", "caf", "missing"] :: [T.Text] + result :: D.DataFrame -> D.Expr Bool -> [Bool] + result df e = DI.toList @Bool (unsafeGetColumn "r" (D.derive "r" e df)) + assertBool + "sorted payload stays PackedText" + (DI.isPackedText (unsafeGetColumn "k" sorted)) + sequence_ + [ assertEqual + (label ++ " " ++ show lit) + (result (boxed df) e) + (result df e) + | (label, df) <- [("base", base), ("sorted", sorted)] + , lit <- lits + , e <- + [ F.col @T.Text "k" .==. F.lit lit + , F.col @T.Text "k" ./=. F.lit lit + , F.lit lit .==. F.col @T.Text "k" + , F.lit lit ./=. F.col @T.Text "k" + ] + ] + tests :: [Test] tests = [ TestLabel "PackedText display parity" displayParity @@ -221,4 +255,5 @@ , TestLabel "PackedText slice preserves packed" slicePreservesPacked , TestLabel "PackedText takeLast preserves packed" takeLastPreservesPacked , TestLabel "PackedText slice rejects bad bounds" sliceRejectsInvalidBounds + , TestLabel "PackedText eq/neq literal kernel parity" eqLiteralKernelParity ] diff --git a/tests/LazyParity.hs b/tests/LazyParity.hs --- a/tests/LazyParity.hs +++ b/tests/LazyParity.hs @@ -2,11 +2,6 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The HARD GATE for the lazy engine: for a bounded source the lazy result -must be BYTE-IDENTICAL to the eager result of the same ops (bounded joins and -aggregates route through the eager fast paths). Covers a LEFT-join pipeline -(join -> groupBy -> aggregate -> sort) and a pure groupBy pipeline. --} module LazyParity (tests) where import qualified Data.Map.Strict as M @@ -48,12 +43,10 @@ , ("plan", schemaType @Text) ] -{- | @n@ orders; ~10% reference a customer id with no matching customer row so -the LEFT join produces a Nothing group (exercises unmatched-row semantics). --} +-- | Orders frame where about 10% reference a customer with no matching row. ordersFrame :: Int -> D.DataFrame ordersFrame n = - D.fromNamedColumns + D.fromColumns [ ("order_id", DI.fromList [0 .. n - 1]) , ( "customer_id" @@ -69,7 +62,7 @@ customersFrame :: Int -> D.DataFrame customersFrame m = - D.fromNamedColumns + D.fromColumns [ ("customer_id", DI.fromList [0 .. m - 1]) , ("region", DI.fromList [T.pack ("r" ++ show (i `mod` 4)) | i <- [0 .. m - 1]]) , ("plan", DI.fromList [T.pack ("p" ++ show (i `mod` 3)) | i <- [0 .. m - 1]]) @@ -84,9 +77,7 @@ removeFile csvPath return r -{- | LEFT join 20k orders to 257 customers -> groupBy region,plan --> sum(amount-discount), count -> sort revenue desc. --} +-- | Left-join orders to customers, then group, aggregate and sort. joinPipelineParity :: Test joinPipelineParity = TestCase $ @@ -113,7 +104,7 @@ |> L.sortBy [("revenue", Descending)] |> L.runDataFrame assertEqual - "join pipeline lazy == eager (byte-identical)" + "join pipeline lazy == eager" (show eager) (show lazy) @@ -138,7 +129,7 @@ |> L.sortBy [("total", Descending)] |> L.runDataFrame assertEqual - "groupBy pipeline lazy == eager (byte-identical)" + "groupBy pipeline lazy == eager" (show eager) (show lazy) @@ -172,7 +163,7 @@ (show actual) where input = - D.fromNamedColumns + D.fromColumns [ ("order_id", DI.fromList [1 .. 12 :: Int]) , ("customer_id", DI.fromList [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2 :: Int]) , diff --git a/tests/LazyParquet.hs b/tests/LazyParquet.hs --- a/tests/LazyParquet.hs +++ b/tests/LazyParquet.hs @@ -87,7 +87,7 @@ assertEqual "filterAndProject" (2, 2) (D.dimensions actual) ) --- Test 5: multi-file glob — alltypes_plain*.parquet matches plain + snappy = 10 rows. +-- Test 5: multi-file glob matches plain + snappy = 10 rows. multiFileGlob :: Test multiFileGlob = TestCase diff --git a/tests/LazyProjection.hs b/tests/LazyProjection.hs --- a/tests/LazyProjection.hs +++ b/tests/LazyProjection.hs @@ -2,9 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | A lazy scan reads only the columns its 'Schema' names, and rejects -self-contradictory 'ReadOptions' before opening the file. --} +-- | A lazy scan reads only the columns its schema names. module LazyProjection (tests) where import qualified Data.Map.Strict as M @@ -42,9 +40,7 @@ twoCols = Schema $ M.fromList [("id", schemaType @Int), ("dob", schemaType @Int)] -{- | The scan keeps only the schema's columns — and 'dob' is not a prefix of -the header, so a positional scheme could not express this. --} +-- | The scan keeps only the schema's columns, whatever their header position. scanProjects :: Test scanProjects = TestLabel "scan_projects" $ TestCase $ withCsv ',' $ \csvPath -> do df <- L.runDataFrame (L.scanCsv twoCols (T.pack csvPath)) @@ -69,14 +65,11 @@ L.runDataFrame (L.scanCsvWith Fast.fastReadCsvWithOpts twoCols (T.pack csvPath)) assertEqual "same frame" slow fast -{- | 'scanSeparated' used to drop its separator on the floor: the reader was -built from the schema alone and always split on commas. --} +-- | scanSeparated must honour its separator, not always split on commas. scanHonoursSeparator :: Test scanHonoursSeparator = TestLabel "scan_honours_separator" $ TestCase $ withCsv ';' $ \csvPath -> do - -- With the separator ignored, the whole line is one field and the - -- schema's columns would not be found at all. + -- With the separator ignored, the schema's columns are not found. df <- L.runDataFrame (L.scanSeparated ';' twoCols (T.pack csvPath)) assertEqual "dims" (2, 2) (D.dimensions df) assertEqual "columns" ["dob", "id"] (D.columnNames df) @@ -141,10 +134,7 @@ "cannot delimit fields" ] -{- | The SIMD reader's low-level entry takes its separator as a scanner byte, -so a 'columnSeparator' saying something else would be silently ignored. -Reject it instead. --} +-- | A column separator that disagrees with the scanner byte is rejected. rejectSeparatorMismatch :: Test rejectSeparatorMismatch = TestLabel "reject_separator_mismatch" $ TestCase $ do let opts = Csv.defaultReadOptions{Csv.columnSeparator = ';'} diff --git a/tests/Learn/Denotation.hs b/tests/Learn/Denotation.hs --- a/tests/Learn/Denotation.hs +++ b/tests/Learn/Denotation.hs @@ -1,12 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The "predict is the model's denotation" claim, discharged as a test rather -than left as a slogan: the @Expr@ that 'predict' compiles must evaluate to the -same numbers as the fitted record's own parameters, computed independently here -in plain Haskell. If @affineExpr@ / @argMinExpr@ ever drift from the record they -are built from, these fail. --} +-- | The Expr from predict must match the fitted parameters numerically. module Learn.Denotation (tests) where import qualified Data.Text as T @@ -22,23 +17,22 @@ import DataFrame.KMeans import DataFrame.LinearModel -import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double] -interpD df e = case interpret @Double df e of +interpD frame e = case interpret @Double frame e of Right (TColumn c) -> either (const []) VU.toList (toVector @Double @VU.Vector c) Left err -> error (show err) interpI :: D.DataFrame -> Expr Int -> [Int] -interpI df e = case interpret @Int df e of +interpI frame e = case interpret @Int frame e of Right (TColumn c) -> either (const []) VU.toList (toVector @Int @VU.Vector c) Left err -> error (show err) df :: D.DataFrame df = - D.fromNamedColumns + D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [3 + 2 * a - 0.5 * b | (a, b) <- zip xs1 xs2]) @@ -50,7 +44,7 @@ col :: D.DataFrame -> T.Text -> [Double] col d n = interpD d (F.col @Double n) --- | @interpret (predict m)@ must equal @intercept + Σ coefⱼ·featureⱼ@ from the record. +-- | @interpret (predict m)@ must equal @intercept + Σ coefⱼ·featureⱼ@. linearDenotation :: Test linearDenotation = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") df @@ -66,7 +60,7 @@ "linear: predict Expr matches the record's affine prediction" (and (zipWith (\a b -> abs (a - b) < 1e-9) native symbolic)) --- | @interpret (predict km)@ must equal the nearest-centroid label from @kmCenters@. +-- | @interpret (predict km)@ must equal the nearest-centroid label. kmeansDenotation :: Test kmeansDenotation = TestCase $ do let feats = ["x1", "x2"] @@ -88,7 +82,7 @@ transposeL [] = [] transposeL xss | any null xss = [] - | otherwise = map head xss : transposeL (map tail xss) + | otherwise = [x | (x : _) <- xss] : transposeL [xs | (_ : xs) <- xss] tests :: [Test] tests = [linearDenotation, kmeansDenotation] diff --git a/tests/Learn/Ensembles.hs b/tests/Learn/Ensembles.hs --- a/tests/Learn/Ensembles.hs +++ b/tests/Learn/Ensembles.hs @@ -19,7 +19,6 @@ import Data.Maybe (isJust, isNothing, listToMaybe) import qualified Data.Vector.Unboxed as VU -import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double] @@ -34,14 +33,14 @@ clsDF :: D.DataFrame clsDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([-3, -2, -1, -0.5, 0.5, 1, 2, 3] :: [Double])) , ("label", DI.fromList ([0, 0, 0, 0, 1, 1, 1, 1] :: [Int])) ] blobs :: D.DataFrame blobs = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList ([0, 0.2, -0.1, 0.1, 8, 8.1, 7.9, 8.2] :: [Double])) , ("b", DI.fromList ([0, -0.1, 0.2, 0.0, 5, 5.2, 4.9, 5.1] :: [Double])) ] @@ -49,7 +48,7 @@ testGBMRegression :: Test testGBMRegression = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1 .. 12] :: [Double])) , ( "y" @@ -73,7 +72,7 @@ testGBMStaged :: Test testGBMStaged = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1 .. 8] :: [Double])) , ("y", DI.fromList ([1 .. 8] :: [Double])) ] @@ -110,7 +109,7 @@ testDBSCAN :: Test testDBSCAN = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList ([0, 0.1, 0.2, 5, 5.1, 5.2, 50] :: [Double])) , ("b", DI.fromList ([0, 0.1, 0.0, 5, 5.0, 5.1, 50] :: [Double])) ] @@ -132,7 +131,7 @@ testGridSearch :: Test testGridSearch = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1 .. 40] :: [Double])) , ( "y" @@ -152,18 +151,10 @@ assertBool "best score high" (gsBestScore res > 0.99) assertEqual "all configs scored" 3 (length (gsAll res)) -{- | Logistic boosting must recover a known conditional probability, not merely -rank it. A leaf set to the mean gradient instead of @Σg/Σh@ understeps every -step by at least 4x, leaving probabilities shrunk toward the base rate while the -ranking — and so any accuracy or AUC check — still looks healthy. - -Each @x@ carries a fixed 20 rows of which exactly @round (20 * trueP x)@ are -positive, so the empirical conditional probability at every @x@ is 'trueP' and -the target is separable in rank but not in value. --} +-- | Logistic boosting must recover a known conditional probability. sigmoidCurveDF :: D.DataFrame sigmoidCurveDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList (concatMap (replicate group . fst) cells)) , ("label", DI.fromList (concatMap snd cells)) ] @@ -199,9 +190,81 @@ ("logistic boosting recovers the conditional probability " ++ show err) (err < 0.03) +gbmData :: D.DataFrame +gbmData = + D.fromColumns + [ ("x", DI.fromList ([fromIntegral i / 4 | i <- [1 .. 40 :: Int]] :: [Double])) + , + ( "z" + , DI.fromList + ([fromIntegral ((i * 7) `mod` 11) | i <- [1 .. 40 :: Int]] :: [Double]) + ) + , + ( "y" + , DI.fromList + ( [ sin (fromIntegral i / 4) + fromIntegral ((i * 7) `mod` 11) * 0.1 + | i <- [1 .. 40 :: Int] + ] :: + [Double] + ) + ) + ] + +-- A constant base equal to the default initial score boosts the same trees. +testGBMBaseScore :: Test +testGBMBaseScore = TestCase $ do + let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2} + ys = interpD gbmData (F.col @Double "y") + mean = sum ys / fromIntegral (length ys) + plain = fit cfg (F.col @Double "y") gbmData + based = fit cfg{gbBaseScore = Just (F.lit mean)} (F.col @Double "y") gbmData + gap = + maximum + ( zipWith + (\a b -> abs (a - b)) + (interpD gbmData (predict plain)) + (interpD gbmData (predict based)) + ) + assertBool + ("base-score model matches the default, gap " ++ show gap) + (gap < 1e-9) + -- A per-row base is part of the prediction: shifting it shifts every score. + let shifted = fit cfg{gbBaseScore = Just (F.col @Double "z")} (F.col @Double "y") gbmData + noTrees = + fit + cfg{gbNEstimators = 0, gbBaseScore = Just (F.col @Double "z")} + (F.col @Double "y") + gbmData + assertEqual + "zero trees predict the base" + (interpD gbmData (F.col @Double "z")) + (interpD gbmData (predict noTrees)) + assertBool + "base model still fits" + (length (interpD gbmData (predict shifted)) == 40) + +-- Sampling at 1 is inert; below 1 it changes the fit, fixed by the seed. +testGBMSubsample :: Test +testGBMSubsample = TestCase $ do + let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2} + scores :: GBConfig -> [Double] + scores c = interpD gbmData (predict (fit c (F.col @Double "y") gbmData)) + assertEqual + "subsample 1 / colsample 1 reproduce the default" + (scores cfg) + (scores cfg{gbSubsample = 1, gbColsample = 1}) + let half = cfg{gbSubsample = 0.5, gbColsample = 0.5, gbSeed = 3} + assertEqual "deterministic in the seed" (scores half) (scores half) + assertBool "sampling changes the fit" (scores half /= scores cfg) + assertBool + "a different seed changes the fit" + (scores half /= scores half{gbSeed = 4}) + tests :: [Test] tests = [ testGBMRegression + , testGBMBaseScore + , testGBMSubsample , testGBMStaged , testGBMCalibration , testAdaBoost diff --git a/tests/Learn/Metamorphic.hs b/tests/Learn/Metamorphic.hs --- a/tests/Learn/Metamorphic.hs +++ b/tests/Learn/Metamorphic.hs @@ -1,15 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Metamorphic, invariance, and abstraction-law tests for the ML library. - -These check how the fitted model and the metrics behave under transformations of -the *input* — duplicate / permute / scale / rename / add-constant-feature — where -the math dictates the answer. Each test computes the model on two genuinely -different (transformed) frames and asserts they agree, or pins a metric/scaler to -its defining law. None compares a value to itself; each can fail if the library -is wrong. --} +-- | Metamorphic and invariance tests: models must obey transformation laws. module Learn.Metamorphic (tests) where import qualified Data.Text as T @@ -33,8 +25,6 @@ import Test.HUnit --- Helpers ------------------------------------------------------------------- - interpD :: D.DataFrame -> Expr Double -> [Double] interpD df e = case interpret @Double df e of Right (TColumn c) -> either (const []) VU.toList (toVector @Double @VU.Vector c) @@ -49,7 +39,7 @@ -- Build a regression frame from explicit feature/target lists. mkRegDF :: [(T.Text, [Double])] -> [Double] -> D.DataFrame mkRegDF feats ys = - D.fromNamedColumns + D.fromColumns ([(n, DI.fromList vs) | (n, vs) <- feats] ++ [("y", DI.fromList ys)]) -- Base design: a noiseless linear target so OLS recovers a unique solution. @@ -64,12 +54,7 @@ fitBase :: D.DataFrame -> LinearRegressor fitBase = fit defaultLinearConfig (F.col @Double "y") --- Metamorphic: duplicate rows ---------------------------------------------- - -{- | OLS minimises Σ(y-ŷ)²; duplicating every row scales the loss by 2 but leaves -the argmin (the coefficients/intercept) unchanged. Catches a solver that -weights rows by frame size or mishandles the normal equations. --} +-- | Duplicating every row leaves the OLS coefficients unchanged. testDuplicateRows :: Test testDuplicateRows = TestCase $ do let dupDF = @@ -85,12 +70,7 @@ "duplicate rows: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Metamorphic: permute rows ------------------------------------------------- - -{- | OLS is invariant to row order. We reverse the rows (a genuine non-identity -permutation) and require identical coefficients. Catches any order-dependence -in matrix assembly or solving. --} +-- | OLS is invariant to row order. testPermuteRows :: Test testPermuteRows = TestCase $ do let permDF = @@ -106,13 +86,7 @@ "permute rows: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Metamorphic: scale a feature ---------------------------------------------- - -{- | Scaling feature x1 by k must scale that coefficient by 1/k while leaving -predictions (and every other coefficient + the intercept) invariant. Both -sides are genuine fits on different frames. Catches a regressor that drops or -mis-applies a feature's scale. --} +-- | Scaling a feature by k scales its coefficient by 1/k and keeps predictions. testScaleFeature :: Test testScaleFeature = TestCase $ do let k = 10.0 @@ -123,7 +97,7 @@ coef1 = VU.toList (regCoef m1) assertBool "scale feature: x1 coefficient scaled by 1/k" - (close 1e-7 (head coef0 / k) (head coef1)) + (close 1e-7 (VU.head (regCoef m0) / k) (VU.head (regCoef m1))) assertBool "scale feature: x2 coefficient unchanged" (close 1e-7 (coef0 !! 1) (coef1 !! 1)) @@ -135,12 +109,7 @@ p1 = interpD scaledDF (predict m1) assertBool "scale feature: predictions invariant" (closeList 1e-6 p0 p1) --- Metamorphic: rename columns ----------------------------------------------- - -{- | Renaming feature columns is a pure relabelling: coefficients, intercept, and -predictions are identical and the fitted feature names track the rename. -Catches any hidden dependence on specific column-name strings. --} +-- | Renaming feature columns only relabels coefficients and feature names. testRenameColumns :: Test testRenameColumns = TestCase $ do let renamedDF = mkRegDF [("feat_a", baseX1), ("feat_b", baseX2)] baseY @@ -161,14 +130,7 @@ p1 = interpD renamedDF (predict m1) assertBool "rename: predictions unchanged" (closeList 1e-9 p0 p1) --- Metamorphic: irrelevant constant feature ---------------------------------- - -{- | A constant feature carries no variance, so OLS predictions for the real -features must be unchanged when one is added. (Its own weight is absorbed into -the intercept and is not separately identifiable, so we pin predictions, not -the weight.) Catches a fit whose answer drifts when a degenerate column is -present. --} +-- | A constant feature must not change predictions for the real features. testConstantFeatureIgnored :: Test testConstantFeatureIgnored = TestCase $ do let withConst = @@ -189,20 +151,12 @@ let coef1 = VU.toList (regCoef m1) assertBool "constant feature: x1 coefficient unchanged" - (close 1e-6 (head (VU.toList (regCoef m0))) (head coef1)) + (close 1e-6 (VU.head (regCoef m0)) (VU.head (regCoef m1))) assertBool "constant feature: x2 coefficient unchanged" (close 1e-6 (VU.toList (regCoef m0) !! 1) (coef1 !! 1)) --- Metamorphic: split then concatenate -------------------------------------- - -{- | Genuinely split the frame into two row-range sub-frames and concatenate them -back with the DataFrame @<>@ (row concatenation). The rebuilt frame has the -same rows, so the fit must be identical — but this actually exercises the -concatenation code path, so it catches a @<>@ that drops, duplicates, or -mis-aligns rows/columns (a plain @take h ++ drop h@ would be the identity and -test nothing). --} +-- | Splitting a frame and rejoining it with <> must give an identical fit. testSplitConcat :: Test testSplitConcat = TestCase $ do let n = length baseX1 @@ -226,14 +180,7 @@ "split/concat: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Training/inference parity: column order ---------------------------------- - -{- | The order of feature columns in the frame must not change predictions. -We fit on a frame whose columns are declared in the opposite order and require -the prediction expression to produce the same numbers on the original frame. -(predict compiles to a name-keyed affine Expr, so it should be order-free.) -Catches a positional, rather than name-keyed, feature mapping. --} +-- | The order of feature columns must not change predictions. testColumnOrderParity :: Test testColumnOrderParity = TestCase $ do let swappedDF = mkRegDF [("x2", baseX2), ("x1", baseX1)] baseY @@ -300,7 +247,7 @@ a0 = accuracy preds truth a1 = accuracy (VU.reverse preds) (VU.reverse truth) assertBool "accuracy: permutation invariant" (close 1e-12 a0 a1) - -- Sanity: the metric is non-trivial here (not 0 or 1), so invariance is meaningful. + -- Sanity: the metric is non-trivial here, so invariance is meaningful. assertBool "accuracy: non-degenerate baseline" (a0 > 0 && a0 < 1) testR2Anchors :: Test @@ -314,9 +261,7 @@ let worse = VU.map (+ 100) truth assertBool "r2: far-off prediction is negative" (r2 worse truth < 0) -{- | r² on a fitted noiseless linear model equals 1, and rmse equals 0. Ties the -metric law to the model: if the fit is exact, the metric must say so. --} +-- | A noiseless linear fit has r2 equal to 1 and rmse equal to 0. testR2OfFittedModel :: Test testR2OfFittedModel = TestCase $ do let m = fitBase baseDF @@ -329,7 +274,7 @@ testTargetNeverAFeature = TestCase $ do let n = 40 :: Int leakDF = - D.fromNamedColumns + D.fromColumns [ ("y", DI.fromList [if even i then 1.0 else 0.0 :: Double | i <- [0 .. n - 1]]) , ("a", DI.fromList [fromIntegral (i `div` 4) :: Double | i <- [0 .. n - 1]]) , ("b", DI.fromList [fromIntegral (i `mod` 3) :: Double | i <- [0 .. n - 1]]) diff --git a/tests/Learn/MetricsTests.hs b/tests/Learn/MetricsTests.hs --- a/tests/Learn/MetricsTests.hs +++ b/tests/Learn/MetricsTests.hs @@ -17,8 +17,10 @@ import DataFrame.PCA import DataFrame.Transform +import Data.List (sortBy) +import Data.Ord (comparing) import qualified Data.Vector.Unboxed as VU -import DataFrame.Model (fit, predict) +import System.Random (mkStdGen, randomRs) import Test.HUnit close :: Double -> Double -> Double -> Bool @@ -30,7 +32,7 @@ reg :: D.DataFrame reg = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1 .. 20] :: [Double])) , ( "y" @@ -117,9 +119,63 @@ assertBool "pipeline produced a frame" (D.columnNames out /= []) assertBool "pipeline has pc1 column" ("pc1" `elem` D.columnNames out) +-- | Mann-Whitney AUC computed from ranks over lists, the reference for rocAuc. +referenceAuc :: [Double] -> [Double] -> Double +referenceAuc scores truth + | nPos == 0 || nNeg == 0 = 0.5 + | otherwise = (rankSum - nPos * (nPos + 1) / 2) / (nPos * nNeg) + where + sorted = sortBy (comparing snd) (zip [0 :: Int ..] scores) + ranks = map snd (sortBy (comparing fst) (assign 1 sorted)) + assign _ [] = [] + assign start grp@((_, v) : _) = + let (tied, rest) = span ((== v) . snd) grp + k = length tied + avg = fromIntegral (sum [start .. start + k - 1]) / fromIntegral k + in [(i, avg) | (i, _) <- tied] ++ assign (start + k) rest + rankSum = sum [r | (y, r) <- zip truth ranks, y == 1] + nPos = fromIntegral (length (filter (== 1) truth)) + nNeg = fromIntegral (length truth) - nPos + +testRocAucMatchesReference :: Test +testRocAucMatchesReference = TestCase $ do + let grid = + [ (n, levels, seed) + | n <- [1, 2, 17, 1000] + , levels <- [2, 7, 100000] + , seed <- [1, 2, 3] + ] + sequence_ + [ assertEqual + ("n=" ++ show n ++ " levels=" ++ show levels ++ " seed=" ++ show seed) + (referenceAuc scores truth) + (rocAuc (VU.fromList scores) (VU.fromList truth)) + | (n, levels, seed) <- grid + , let scores = map fromIntegral (take n (randomRs (0, levels - 1 :: Int) (mkStdGen seed))) + truth = map fromIntegral (take n (randomRs (0, 1 :: Int) (mkStdGen (seed + 100)))) + ] + +testStratifiedFolds :: Test +testStratifiedFolds = TestCase $ do + let labels = + VU.fromList (map fromIntegral (take 1003 (randomRs (0, 1 :: Int) (mkStdGen 7)))) + folds = stratifiedFoldIds 5 42 labels + perFold c = + [ VU.length (VU.filter id (VU.zipWith (\f l -> f == k && l == c) folds labels)) + | k <- [0 .. 4] + ] + spread xs = maximum xs - minimum xs + assertEqual "one fold per row" (VU.length labels) (VU.length folds) + assertBool "folds in range" (VU.all (\f -> f >= 0 && f < 5) folds) + assertBool "positives balanced" (spread (perFold 1) <= 1) + assertBool "negatives balanced" (spread (perFold 0) <= 1) + assertEqual "deterministic" folds (stratifiedFoldIds 5 42 labels) + tests :: [Test] tests = - [ testRegressionMetrics + [ testRocAucMatchesReference + , testStratifiedFolds + , testRegressionMetrics , testMulticlassMetrics , testRocAuc , testReports diff --git a/tests/Learn/Models.hs b/tests/Learn/Models.hs --- a/tests/Learn/Models.hs +++ b/tests/Learn/Models.hs @@ -21,11 +21,9 @@ import Control.Exception (evaluate, try) import Data.List (isInfixOf) import qualified Data.Map.Strict as M -import qualified Data.Text as T import qualified Data.Vector as V import qualified Data.Vector.Unboxed as VU import DataFrame.Errors (DataFrameException) -import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double] @@ -43,7 +41,7 @@ regDF :: D.DataFrame regDF = - D.fromNamedColumns + D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2]) @@ -57,7 +55,7 @@ flowerDF :: D.DataFrame flowerDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([-5, -4, -3, 0, 0.3, 0.6, 5, 6, 7] :: [Double])) , ( "flower" @@ -67,7 +65,7 @@ clsDF :: D.DataFrame clsDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([-3, -2, -1, -0.5, 0.5, 1, 2, 3] :: [Double])) , ("label", DI.fromList ([0, 0, 0, 0, 1, 1, 1, 1] :: [Int])) ] @@ -83,19 +81,15 @@ truth = interpD regDF (F.col @Double "y") assertBool "OLS expr matches y" (and (zipWith (close 1e-6) preds truth)) -{- | A regression frame whose feature column @x@ is stored as @Int@; the target -@y@ is @Double@. Fitting must reject the Int feature at the boundary. --} +-- | Regression frame whose feature x is stored as Int. intFeatureDF :: D.DataFrame intFeatureDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList @Int [1 .. 8]) , ("y", DI.fromList @Double [2, 4, 6, 8, 10, 12, 14, 16]) ] -{- | The fit boundary mandates Double: a non-Double feature column is a fit-time -error (naming the column and pointing at the fix), not a silent coercion. --} +-- | A non-Double feature column is a fit-time error, not a silent coercion. testFitRequiresDouble :: Test testFitRequiresDouble = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") intFeatureDF @@ -108,19 +102,15 @@ assertBool "error names the offending column x" ("x" `isInfixOf` msg) Right _ -> assertFailure "expected a Double-input error for the Int feature column" -{- | A regression frame whose feature @x@ is nullable (@Maybe Double@). Fitting -must reject it rather than letting nulls poison the solve into NaN. --} +-- | Regression frame whose feature x is nullable. nullableFeatureDF :: D.DataFrame nullableFeatureDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList @(Maybe Double) (Nothing : map Just [2, 3, 4, 5, 6, 7, 8])) , ("y", DI.fromList @Double [2, 4, 6, 8, 10, 12, 14, 16]) ] -{- | The fit boundary rejects a nullable (Maybe Double) input column with an -actionable error, instead of silently producing NaN coefficients. --} +-- | Fitting rejects a nullable input column with an actionable error. testFitRejectsNullable :: Test testFitRejectsNullable = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") nullableFeatureDF @@ -153,9 +143,7 @@ let probs = logisticProbExprs m assertBool "prob exprs present for both classes" (M.size probs == 2) -{- | Logistic regression must work with any enum label type, not just Int — -multiclass classification into a user-defined sum type. --} +-- | Logistic regression must work with any enum label type, not just Int. testLogisticEnumLabel :: Test testLogisticEnumLabel = TestCase $ do let m = fit defaultLogisticConfig (F.col @Flower "flower") flowerDF @@ -206,14 +194,14 @@ testKMeans :: Test testKMeans = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList ([0, 0.1, 0.2, 10, 10.1, 10.2] :: [Double])) , ("b", DI.fromList ([0, 0.1, 0, 10, 10, 10.1] :: [Double])) ] cfg = defaultKMeansConfig{kmK = 2, kmNInit = 5, kmSeed = 1} m = fit cfg [F.col @Double "a", F.col @Double "b"] df labels = VU.toList (kmLabels m) - assertBool "two blobs split" (head labels /= last labels) + assertBool "two blobs split" (VU.head (kmLabels m) /= VU.last (kmLabels m)) assertEqual "two centers" 2 (V.length (kmCenters m)) let m2 = fit cfg [F.col @Double "a", F.col @Double "b"] df assertEqual "kmeans deterministic" (kmCenters m) (kmCenters m2) diff --git a/tests/Learn/Segmented.hs b/tests/Learn/Segmented.hs --- a/tests/Learn/Segmented.hs +++ b/tests/Learn/Segmented.hs @@ -44,7 +44,7 @@ -- | A two-segment frame: y = 2x+1 on group "a", y = -3x+5 on group "b". twoSegDF :: D.DataFrame twoSegDF = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("x", DI.fromList xs) , @@ -57,7 +57,9 @@ segByKey :: [T.Text] -> SegmentedModel a LinearRegressor -> Segment LinearRegressor -segByKey k m = head [s | s <- smSegments m, segKey s == k] +segByKey k m = case [s | s <- smSegments m, segKey s == k] of + (s : _) -> s + [] -> error ("segByKey: no segment for " ++ show k) lowFloor :: Segmented cfg -> Segmented cfg lowFloor s = s{segMinRows = 3} @@ -86,7 +88,7 @@ let m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") twoSegDF fb = smFallback m unseen = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (["c"] :: [T.Text])) , ("x", DI.fromList ([2.0] :: [Double])) , ("y", DI.fromList ([0.0] :: [Double])) @@ -103,7 +105,7 @@ [ "logistic base composes" ~: do let labels = replicate 4 ("pos" :: T.Text) ++ replicate 4 "neg" df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double])) , ("label", DI.fromList labels) @@ -122,7 +124,7 @@ selectionTests = [ "segmentOn picks only the named column" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("h", DI.fromList (cycle3 8)) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double])) @@ -136,7 +138,7 @@ assertEqual "only g" ["g"] (smCatCols m) , "cardinality cap skips high-cardinality text" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("k", DI.fromList (map (T.pack . show) [1 .. 8 :: Int])) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double])) @@ -156,7 +158,7 @@ [Just "a", Just "a", Just "a", Nothing, Just "b", Just "b", Just "b", Nothing] :: [Maybe T.Text] df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList gcol) , ("x", DI.fromList ([1, 2, 3, 9, 1, 2, 3, 9] :: [Double])) , ("y", DI.fromList ([3, 5, 7, 0, 2, -1, -4, 0] :: [Double])) @@ -180,7 +182,7 @@ guardTests = [ "int feature error points at toDouble" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("n", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Int])) , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double])) @@ -194,7 +196,7 @@ Right _ -> assertFailure "expected an actionable error for the Int feature" , "nullable feature error points at dropping" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ( "z" @@ -214,7 +216,7 @@ Right _ -> assertFailure "expected an actionable error for the nullable feature" , "well-typed frame fits after casting/dropping" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("n", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double])) , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double])) @@ -227,7 +229,7 @@ diagnosticTests = [ "undersized segment falls back, diagnostics correct" ~: do let df = - D.fromNamedColumns + D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ ["b"])) , ("x", DI.fromList ([1, 2, 3, 4, 9] :: [Double])) , ("y", DI.fromList ([3, 5, 7, 9, 0] :: [Double])) @@ -273,15 +275,14 @@ -- lambda = 0: independent OLS (matches recovery test). assertBool "lambda0 a slope 2" (close 1e-9 (slope ["a"] m0) 2) assertBool "lambda0 b slope -3" (close 1e-9 (slope ["b"] m0) (-3)) - -- lambda -> inf: both segments converge to the n_g-weighted reference - -- (raw slope -0.5, raw intercept 3.0 for this balanced fixture). + -- lambda -> inf: both segments converge to the weighted reference. assertBool "lambdaInf a slope ~ -0.5" (close 1e-3 (slope ["a"] mInf) (-0.5)) assertBool "lambdaInf b slope ~ -0.5" (close 1e-3 (slope ["b"] mInf) (-0.5)) assertBool "lambdaInf a intercept ~ 3" (close 1e-3 (inter ["a"] mInf) 3.0) assertBool "lambdaInf segments coincide" (close 1e-3 (slope ["a"] mInf) (slope ["b"] mInf)) - -- interior: a's slope sits strictly between its OLS (2) and the ref (-0.5). + -- interior: a's slope sits strictly between its OLS and the ref. assertBool "interior between OLS and ref" (slope ["a"] mMid < 2 && slope ["a"] mMid > (-0.5)) diff --git a/tests/Learn/SklearnParity.hs b/tests/Learn/SklearnParity.hs --- a/tests/Learn/SklearnParity.hs +++ b/tests/Learn/SklearnParity.hs @@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Parity tests against scikit-learn on clean Kaggle-style datasets; reference -values in @data/ml/golden.json@ come from @scripts/gen_sklearn_golden.py@. -Closed-form models get tight coefficient parity; iterative ones an accuracy floor. --} +-- | Parity tests against scikit-learn using reference values in golden.json. module Learn.SklearnParity (tests) where import Data.Aeson (Value (..), decodeFileStrict') @@ -14,7 +11,6 @@ import qualified Data.Text as T import qualified Data.Vector as V import qualified Data.Vector.Unboxed as VU -import DataFrame.Model (fit, predict) import Test.HUnit import qualified DataFrame as D diff --git a/tests/Learn/Synthesis.hs b/tests/Learn/Synthesis.hs --- a/tests/Learn/Synthesis.hs +++ b/tests/Learn/Synthesis.hs @@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Feature-synthesis enumerator: it should recover small exact features -(@x²@, @a/b@) from the example rows, score them near-perfectly, and be -deterministic. --} +-- | The synthesis enumerator recovers small exact features deterministically. module Learn.Synthesis (tests) where import Assertions (assertExpectException) @@ -16,7 +13,7 @@ quad :: D.DataFrame quad = - D.fromNamedColumns + D.fromColumns [ ("x", D.fromList xs) , ("y", D.fromList (map (\x -> x * x) xs)) ] @@ -25,7 +22,7 @@ ratio :: D.DataFrame ratio = - D.fromNamedColumns + D.fromColumns [ ("a", D.fromList ([2, 6, 12, 20, 30, 42] :: [Double])) , ("b", D.fromList ([1, 2, 3, 4, 5, 6] :: [Double])) , ("y", D.fromList ([2, 3, 4, 5, 6, 7] :: [Double])) @@ -74,11 +71,7 @@ (D.prettyPrint (sfExpr a)) (D.prettyPrint (sfExpr b)) -{- | A wide frame at the default 'synMaxSize' refuses instead of exhausting the -heap. 'synBankCap' caps what is kept, not what is generated, so the layers past -size 4 used to allocate tens of gigabytes and kill the process — which no test -can catch, because there is no process left to fail. --} +-- | A wide frame at the default synMaxSize refuses, not exhausts the heap. refusesOversizedSearch :: Test refusesOversizedSearch = TestCase @@ -100,7 +93,7 @@ -- | 12 features over 3000 rows: the shape that killed the kernel. wide :: D.DataFrame wide = - D.fromNamedColumns + D.fromColumns ( ("y", D.fromList (map (\i -> fromIntegral (i `mod` 7) :: Double) idx)) : [ ( "f" <> T.pack (show c) , D.fromList (map (\i -> fromIntegral ((i * c) `mod` 13) :: Double) idx) diff --git a/tests/Learn/TypedModel.hs b/tests/Learn/TypedModel.hs --- a/tests/Learn/TypedModel.hs +++ b/tests/Learn/TypedModel.hs @@ -3,11 +3,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | 'fit'/'predict' are frame-aware. Training on a 'TypedDataFrame' with a typed -target ('DT.col') returns a 'Fitted' carrying the schema, and 'predict' on it -yields a typed 'TExpr'; training on an untyped 'DataFrame' with an untyped target -('F.col') returns the bare model and an 'Expr'. Both paths must agree numerically. --} +-- | Typed and untyped fit/predict paths must agree numerically. module Learn.TypedModel (tests) where import Data.Maybe (fromJust) @@ -40,7 +36,7 @@ regDF :: D.DataFrame regDF = - D.fromNamedColumns + D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2]) @@ -65,9 +61,7 @@ and (zipWith (close 1e-12) (VU.toList (regCoef a)) (VU.toList (regCoef b))) && close 1e-12 (regIntercept a) (regIntercept b) -{- | 'fit' on a 'TypedDataFrame' returns a 'Fitted' whose model matches the -bare model from 'fit' on the equivalent 'DataFrame'. --} +-- | Typed fit returns a model matching the fit on the equivalent DataFrame. testTypedFitParity :: Test testTypedFitParity = TestCase $ do let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF @@ -76,14 +70,12 @@ "fittedModel of typed fit == untyped fit" (sameModel (fittedModel onTyped) onUntyped) -{- | 'predict' on the typed fit yields a 'TExpr Houses Double'; unwrapped and -interpreted it matches 'predict' on the untyped fit. --} +-- | Typed predict matches predict on the untyped fit once interpreted. testTypedPredict :: Test testTypedPredict = TestCase $ do let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF onUntyped = fit defaultLinearConfig (F.col @Double "y") regDF - typedExpr = DT.unTExpr (predict onTyped) -- :: Expr Double via TExpr Houses Double + typedExpr = DT.unTExpr (predict onTyped) untypedExpr = predict onUntyped -- :: Expr Double truth = interpD regDF (F.col @Double "y") assertBool diff --git a/tests/Main.hs b/tests/Main.hs --- a/tests/Main.hs +++ b/tests/Main.hs @@ -34,13 +34,17 @@ import qualified Operations.Aggregations import qualified Operations.Apply import qualified Operations.Core +import qualified Operations.CountDistinct import qualified Operations.Derive +import qualified Operations.ExprParse import qualified Operations.Filter import qualified Operations.GroupBy import qualified Operations.Inference import qualified Operations.InsertColumn +import qualified Operations.InterpreterKernels import qualified Operations.Join import qualified Operations.Merge +import qualified Operations.MonomorphicLoops import qualified Operations.Nullable import qualified Operations.NullableHashing import qualified Operations.ParallelGroupBy @@ -48,12 +52,16 @@ import qualified Operations.Provenance import qualified Operations.ReadCsv import qualified Operations.Record +import qualified Operations.RowWise import qualified Operations.SetOps import qualified Operations.Shuffle import qualified Operations.Sort +import qualified Operations.SortMask import qualified Operations.Statistics import qualified Operations.Subset import qualified Operations.Take +import qualified Operations.TextKernels +import qualified Operations.TopBy import qualified Operations.Typing import qualified Operations.VectorKernel import qualified Operations.Window @@ -83,9 +91,14 @@ ++ Learn.SklearnParity.tests ++ Learn.Synthesis.tests ++ Learn.MetricsTests.tests + ++ Operations.ExprParse.tests ++ Learn.Metamorphic.tests ++ Learn.Segmented.tests ++ Operations.Aggregations.tests + ++ Operations.CountDistinct.tests + ++ Operations.SortMask.tests + ++ Operations.TextKernels.tests + ++ Operations.TopBy.tests ++ Operations.Apply.tests ++ Operations.Core.tests ++ Operations.Derive.tests @@ -112,6 +125,9 @@ ++ Operations.Take.tests ++ Operations.Typing.tests ++ Operations.VectorKernel.tests + ++ Operations.InterpreterKernels.tests + ++ Operations.MonomorphicLoops.tests + ++ Operations.RowWise.tests ++ Operations.Window.tests ++ Functions.tests ++ IO.CSV.tests diff --git a/tests/Monad.hs b/tests/Monad.hs --- a/tests/Monad.hs +++ b/tests/Monad.hs @@ -30,7 +30,7 @@ let finalRowCount = D.nRows finalDf let realRate = roundToTwoPlaces $ fromIntegral finalRowCount / fromIntegral rowCount let diff = abs $ expectedRate - realRate - -- calculates the 99.99% confidence interval (quickcheck runs 100 tests, aim for 1/10000) + -- 99.99% confidence interval (quickcheck runs 100 tests, 1/10000) let tolerance' = 3.89 * sqrt (expectedRate * (1 - expectedRate) / fromIntegral rowCount) assert (diff <= tolerance') @@ -41,7 +41,7 @@ verbFixture :: DataFrame verbFixture = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList ([3, 1, 2] :: [Int])) , ("B", DI.fromList (["x", "y", "z"] :: [T.Text])) , ("C", DI.fromList ([1.0, 2.0, 3.0] :: [Double])) diff --git a/tests/Operations/Aggregations.hs b/tests/Operations/Aggregations.hs --- a/tests/Operations/Aggregations.hs +++ b/tests/Operations/Aggregations.hs @@ -25,14 +25,14 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values foldAggregation :: Test foldAggregation = TestCase ( assertEqual "Counting elements after grouping gives correct numbers" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6 :: Int, 3, 3]) ] @@ -49,7 +49,7 @@ TestCase ( assertEqual "countAll gives per-group row counts without a column argument" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("n", DI.fromList [6 :: Int, 3, 3]) ] @@ -66,7 +66,7 @@ TestCase ( assertEqual "Typed countAll gives per-group row counts" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("n", DI.fromList [6 :: Int, 3, 3]) ] @@ -93,7 +93,7 @@ TestCase ( assertEqual "Typed counting elements after grouping gives correct numbers" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2_count", DI.fromList [6 :: Int, 3, 3]) ] @@ -120,7 +120,7 @@ TestCase ( assertEqual "Mean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6.5 :: Double, 8.0, 5.0]) ] @@ -137,7 +137,7 @@ TestCase ( assertEqual "Typed ean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2_mean", DI.fromList [6.5 :: Double, 8.0, 5.0]) ] @@ -164,7 +164,7 @@ TestCase ( assertEqual "Mean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6.5 :: Double, 8.0, 5.0]) ] @@ -183,7 +183,7 @@ TestCase ( assertEqual "Mean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [13 :: Double, 16, 10]) ] @@ -200,7 +200,7 @@ TestCase ( assertEqual "Mean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [12 :: Double, 9, 6]) ] @@ -220,7 +220,7 @@ TestCase ( assertEqual "Mean works for ints" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [24 :: Int, 18, 12]) ] @@ -238,7 +238,7 @@ TestCase ( assertEqual "Aggregation on DataFrame with no rows" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList ([] :: [Int])) , ("sum(test2)", DI.fromList ([] :: [Int])) ] @@ -261,7 +261,7 @@ 3 ( D.nRows ( D.distinct - ( D.fromNamedColumns + ( D.fromColumns [ ("x", DI.fromList [1 :: Int, 1, 2, 2, 3]) , ("y", DI.fromList [10 :: Int, 10, 20, 20, 30]) ] @@ -278,7 +278,7 @@ 3 ( D.nRows ( D.distinct - ( D.fromNamedColumns + ( D.fromColumns [ ("x", DI.fromList [1 :: Int, 2, 3]) , ("y", DI.fromList [10 :: Int, 20, 30]) ] @@ -295,17 +295,17 @@ 1 ( D.nRows ( D.distinct - ( D.fromNamedColumns + ( D.fromColumns [("x", DI.fromList [42 :: Int, 42, 42, 42])] ) ) ) ) --- groupBy on an Optional (nullable) column: Nothing values form their own group. +-- groupBy on a nullable column: Nothing values form their own group. optGroupByDf :: D.DataFrame optGroupByDf = - D.fromNamedColumns + D.fromColumns [ ("key", DI.fromList [Just 1 :: Maybe Int, Just 1, Just 2, Nothing, Nothing]) , ("val", DI.fromList [10 :: Int, 20, 30, 40, 50]) ] diff --git a/tests/Operations/Apply.hs b/tests/Operations/Apply.hs --- a/tests/Operations/Apply.hs +++ b/tests/Operations/Apply.hs @@ -34,7 +34,7 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values applyBoxedToUnboxed :: Test applyBoxedToUnboxed = @@ -93,14 +93,14 @@ ) where nullableApplyData = - D.fromNamedColumns [("test", DI.fromList ["", "b" :: String])] + D.fromColumns [("test", DI.fromList ["", "b" :: String])] applyManyOnlyGivenFields :: Test applyManyOnlyGivenFields = TestCase ( assertEqual "Applies function to many fields" - ( D.fromNamedColumns + ( D.fromColumns ( map (,D.fromList $ replicate 26 (1 :: Integer)) ["test4", "test6"] ++ -- All other fields should have their original values. @@ -119,7 +119,7 @@ TestCase ( assertEqual "Applies function to many fields" - ( D.fromNamedColumns + ( D.fromColumns (map (,D.fromList $ replicate 26 (1 :: Integer)) ["test4", "test6", "test8"]) ) ( D.select ["test4", "test6", "test8"] $ @@ -135,7 +135,7 @@ TestCase ( assertEqual "Unboxes fields when necessary" - ( D.fromNamedColumns + ( D.fromColumns (map (,D.fromList $ replicate 26 (1 :: Int)) ["test4", "test6", "test8"]) ) ( D.select ["test4", "test6", "test8"] $ @@ -229,7 +229,7 @@ imputeData :: D.DataFrame imputeData = - D.fromNamedColumns + D.fromColumns [ ("opt", DI.fromList [Just (1 :: Int), Nothing, Just 3]) , ("plain", DI.fromList [10 :: Int, 20, 30]) ] diff --git a/tests/Operations/Core.hs b/tests/Operations/Core.hs --- a/tests/Operations/Core.hs +++ b/tests/Operations/Core.hs @@ -13,7 +13,7 @@ testData :: D.DataFrame testData = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList ([1 .. 3] :: [Int])) , ("B", DI.fromList ['a' .. 'c']) ] @@ -70,15 +70,13 @@ TestCase ( assertEqual "null cell preserves row alignment" - ( D.fromNamedColumns + ( D.fromColumns [("A", DI.fromList ([Just 1, Nothing, Just 3] :: [Maybe Int]))] ) (D.fromRows ["A"] [[D.toAny (1 :: Int)], [Null], [D.toAny (3 :: Int)]]) ) -{- | An all-null column has as many rows as it was given. Collapsing it to an -empty column silently truncates the frame. --} +-- | An all-null column keeps the row count it was given. fromRowsAllNullColumnKeepsRows :: Test fromRowsAllNullColumnKeepsRows = TestCase @@ -88,14 +86,12 @@ (D.nRows (D.fromRows ["A"] [[Null], [Null], [Null]])) ) -{- | A frame with a null survives the round trip at full length. Guards the -alignment invariant through 'toRowList' as well as 'fromRows'. --} +-- | A frame with a null survives the round trip at full length. fromRowsRoundTripsWithNulls :: Test fromRowsRoundTripsWithNulls = TestCase ( let df = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList ([Just 1, Nothing, Just 3] :: [Maybe Int])) , ("B", DI.fromList (["x", "y", "z"] :: [T.Text])) ] diff --git a/tests/Operations/CountDistinct.hs b/tests/Operations/CountDistinct.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/CountDistinct.hs @@ -0,0 +1,213 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE TypeApplications #-} + +module Operations.CountDistinct where + +import qualified Data.ByteString as BS +import qualified Data.Map.Strict as M +import qualified Data.Set as S +import qualified Data.Text as T +import qualified Data.Text.Array as A +import qualified Data.Text.Encoding as TE +import qualified Data.Vector.Unboxed as VU +import qualified DataFrame as D +import qualified DataFrame.Functions as F +import qualified DataFrame.Internal.Column as DI + +import Data.Bits (shiftR, xor) +import Data.Function ((&)) +import Data.Int (Int32, Int64) +import Data.Text.Internal (Text (Text)) +import DataFrame.Expression.Operators (as) +import DataFrame.Internal.Aggregation.Kernel.Distinct ( + countDistinctColumn, + countDistinctGroups, + ) +import DataFrame.Internal.Data.PackedText (mkPackedContiguous) +import Test.HUnit + +-- | Deterministic pseudo-random stream (splitmix-style). +stream :: Int -> Int -> [Int] +stream seed n = take n (drop 1 (map mix (iterate (+ (-7046029254386353131)) seed))) + where + mix z0 = + let z1 = (z0 `xor` (z0 `shiftR` 30)) * (-4658895280553007687) + z2 = (z1 `xor` (z1 `shiftR` 27)) * (-7723592293110705685) + in z2 `xor` (z2 `shiftR` 31) + +-- | Values in @[0, range)@. +draws :: Int -> Int -> Int -> [Int] +draws seed range n = map (`mod` range) (stream seed n) + +refGlobal :: (Ord a) => [a] -> Int +refGlobal = S.size . S.fromList + +refGrouped :: (Ord a) => [Int] -> [a] -> Int -> [Int] +refGrouped gs xs nGroups = + let m = M.fromListWith S.union [(g, S.singleton x) | (g, x) <- zip gs xs] + in [maybe 0 S.size (M.lookup g m) | g <- [0 .. nGroups - 1]] + +-- | Packed UTF-8 column of the given texts. +packed :: [T.Text] -> DI.Column +packed ts = + let bytes = BS.concat (map TE.encodeUtf8 ts) + offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts) + Text arr _ _ = TE.decodeUtf8 bytes + in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs)) + +word :: Int -> T.Text +word k = "w" <> T.pack (show k) + +globalCase :: String -> DI.Column -> Int -> Test +globalCase label col expected = + TestCase (assertEqual label (Just expected) (countDistinctColumn col)) + +groupedCase :: String -> [Int] -> Int -> DI.Column -> [Int] -> Test +groupedCase label gs nGroups col expected = + TestCase + ( assertEqual + label + (Just (VU.fromList expected)) + (countDistinctGroups (VU.fromList gs) nGroups col) + ) + +-- Sizes straddle the parallel threshold and force table growth. +sizes :: [Int] +sizes = [0, 1, 17, 5000, 300000] + +intGlobal :: [Test] +intGlobal = + [ globalCase + ("Int n=" ++ show n ++ " r=" ++ show r) + (DI.fromList xs) + (refGlobal xs) + | n <- sizes + , r <- [3, n `div` 3 + 1, maxBound] + , let xs = draws n r n + ] + +-- 'minBound' is the kernel's empty-slot marker, so it must still count once. +intSentinel :: Test +intSentinel = + let xs = take 300000 (cycle [minBound, 0, -1, maxBound, minBound, 7 :: Int]) + in globalCase "Int with minBound" (DI.fromList xs) (refGlobal xs) + +narrowInts :: [Test] +narrowInts = + [ let xs = map fromIntegral (draws 1 100000 250000) :: [Int64] + in globalCase "Int64" (DI.fromList xs) (refGlobal xs) + , let xs = map fromIntegral (draws 2 100000 250000) :: [Int32] + in globalCase "Int32" (DI.fromList xs) (refGlobal xs) + ] + +doubles :: Test +doubles = + let xs = + [-0.0, 0.0, 1.5, 1.5, -2.25, 1 / 0, -(1 / 0)] + ++ map fromIntegral (draws 3 50 1000) :: + [Double] + in globalCase "Double, -0.0 joins 0.0" (DI.fromList xs) (refGlobal xs) + +texts :: [Test] +texts = + [ globalCase ("Text boxed n=" ++ show n) (DI.fromList ts) (refGlobal ts) + | n <- sizes + , let ts = map word (draws 4 (n `div` 4 + 1) n) + ] + ++ [ globalCase ("Text packed n=" ++ show n) (packed ts) (refGlobal ts) + | n <- sizes + , let ts = map word (draws 5 (n `div` 4 + 1) n) ++ ["", "", "é", "é"] + ] + +grouped :: [Test] +grouped = + [ groupedCase + ("Int grouped n=" ++ show n ++ " g=" ++ show ng) + gs + ng + (DI.fromList xs) + (refGrouped gs xs ng) + | n <- sizes + , ng <- [1, 7, 1000] + , let gs = draws (n + 11) ng n + , let xs = draws (n + 13) (n `div` 2 + 1) n + ] + ++ [ groupedCase "Text grouped" gs 50 (packed ts) (refGrouped gs ts 50) + | let n = 300000 + , let gs = draws 21 50 n + , let ts = map word (draws 22 20000 n) + ] + -- Enough groups to take the shared atomic-count path. + ++ [ groupedCase + "Int grouped, wide domain" + gs + ng + (DI.fromList xs) + (refGrouped gs xs ng) + | let n = 1500000 + , let ng = 600000 + , let gs = draws 31 ng n + , let xs = draws 32 4 n + ] + -- A group id absent from rowToGroup reports zero. + ++ [groupedCase "empty group" [0, 0, 2] 3 (DI.fromList [5, 5, 6 :: Int]) [1, 0, 1]] + +-- The public API: aggregate after groupBy dispatches to the kernel. +viaAggregate :: Test +viaAggregate = + let n = 250000 + gs = draws 41 37 n + xs = draws 42 5000 n + df = D.fromColumns [("g", DI.fromList gs), ("x", DI.fromList xs)] + expected = + D.fromColumns + [ ("g", DI.fromList [0 .. 36 :: Int]) + , ("u", DI.fromList (refGrouped gs xs 37)) + ] + in TestCase + ( assertEqual + "aggregate countDistinct" + expected + ( df + & D.groupBy ["g"] + & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"] + & D.sortBy [D.Asc (F.col @Int "g")] + ) + ) + +-- An Int32 key takes the direct grouping path and must match the reference. +viaAggregateInt32Key :: Test +viaAggregateInt32Key = + let n = 250000 + gs = draws 43 37 n + xs = draws 44 5000 n + df = + D.fromColumns + [("g", DI.fromList (map fromIntegral gs :: [Int32])), ("x", DI.fromList xs)] + expected = + D.fromColumns + [ ("g", DI.fromList [0 .. 36 :: Int32]) + , ("u", DI.fromList (refGrouped gs xs 37)) + ] + in TestCase + ( assertEqual + "aggregate countDistinct, Int32 key" + expected + ( df + & D.groupBy ["g"] + & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"] + & D.sortBy [D.Asc (F.col @Int32 "g")] + ) + ) + +tests :: [Test] +tests = + map + (TestLabel "countDistinct") + ( intGlobal + ++ [intSentinel, doubles, viaAggregate, viaAggregateInt32Key] + ++ narrowInts + ++ texts + ++ grouped + ) diff --git a/tests/Operations/Derive.hs b/tests/Operations/Derive.hs --- a/tests/Operations/Derive.hs +++ b/tests/Operations/Derive.hs @@ -23,7 +23,7 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values deriveWAI :: Test deriveWAI = diff --git a/tests/Operations/ExprParse.hs b/tests/Operations/ExprParse.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/ExprParse.hs @@ -0,0 +1,85 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeApplications #-} + +-- | Reloading expressions from their two printed forms. +module Operations.ExprParse (tests) where + +import qualified Data.Map.Strict as M +import qualified Data.Text as T +import qualified Data.Vector.Unboxed as VU +import qualified DataFrame as D +import DataFrame.Expression.Parse +import qualified DataFrame.Functions as F +import DataFrame.Internal.Expression (Expr, eqExpr, prettyPrint) +import DataFrame.Internal.Expression.Operators ((./=.), (.<=.)) +import Test.HUnit + +frame :: D.DataFrame +frame = + D.fromColumns + [ ("Age", D.fromList [25, 41, 63, 18 :: Int]) + , ("Departure/Arrival time convenient", D.fromList [0, 3, 5, 2 :: Int]) + , ("Class", D.fromList ["Business", "Eco", "Eco Plus", "Eco" :: T.Text]) + , ("Arrival Delay in Minutes", D.fromList [0, 12.5, 3, 0 :: Double]) + ] + +-- A fitted-tree shape: cast threshold, Text inequality, operator-char name. +model :: Expr Double +model = + F.lit 0.1 + * F.ifThenElse + (F.toDouble (F.col @Int "Age") .<=. F.lit 40.5) + ( F.ifThenElse + (F.col @T.Text "Class" ./=. F.lit "Business") + (F.lit (-0.859)) + (F.lit 0.31) + ) + ( F.ifThenElse + (F.toDouble (F.col @Int "Departure/Arrival time convenient") .<=. F.lit 2.5) + (F.lit 1.2) + (F.col @Double "Arrival Delay in Minutes" / F.lit 10) + ) + + F.lit 0.1 + * F.ifThenElse + (F.col @T.Text "Class" ./=. F.lit "Eco") + (F.lit 0.5) + (F.lit (-0.25)) + +scores :: Expr Double -> VU.Vector Double +scores e = + either + (error . show) + id + (D.columnAsUnboxedVector (F.col @Double "s") (D.derive "s" e frame)) + +testShowRoundTrip :: Test +testShowRoundTrip = TestCase $ case parseShownAs @Double (show model) of + Left err -> assertFailure ("parseShown failed: " ++ err) + Right e -> do + assertBool "structurally equal" (eqExpr e model) + assertEqual "same scores" (scores model) (scores e) + +testPrettyRoundTrip :: Test +testPrettyRoundTrip = TestCase $ case parsePrettyAs @Double (schemaOf frame) (prettyPrint model) of + Left err -> assertFailure ("parsePretty failed: " ++ err) + Right e -> do + assertEqual "reprint identical" (prettyPrint model) (prettyPrint e) + let gap = VU.maximum (VU.zipWith (\a b -> abs (a - b)) (scores model) (scores e)) + assertBool ("scores within 1e-12, got " ++ show gap) (gap < 1e-12) + +testSchema :: Test +testSchema = TestCase $ do + let s = schemaOf frame + assertEqual "Int column" (Just TInt) (M.lookup "Age" s) + assertEqual "Text column" (Just TText) (M.lookup "Class" s) + assertEqual + "Double column" + (Just TDouble) + (M.lookup "Arrival Delay in Minutes" s) + +tests :: [Test] +tests = + [ TestLabel "expr parse: show form round-trips" testShowRoundTrip + , TestLabel "expr parse: pretty form round-trips" testPrettyRoundTrip + , TestLabel "expr parse: schema types" testSchema + ] diff --git a/tests/Operations/Filter.hs b/tests/Operations/Filter.hs --- a/tests/Operations/Filter.hs +++ b/tests/Operations/Filter.hs @@ -25,7 +25,7 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values filterColumnDoesNotExist :: Test filterColumnDoesNotExist = @@ -86,10 +86,10 @@ TestCase ( assertEqual "Filters out Nothing and unwraps Maybe" - (D.fromNamedColumns [("test", D.fromList $ replicate 5 (1 :: Int))]) + (D.fromColumns [("test", D.fromList $ replicate 5 (1 :: Int))]) ( D.filterJust "test" - ( D.fromNamedColumns + ( D.fromColumns [("test", D.fromList $ take 10 $ cycle [Just (1 :: Int), Nothing])] ) ) diff --git a/tests/Operations/GroupBy.hs b/tests/Operations/GroupBy.hs --- a/tests/Operations/GroupBy.hs +++ b/tests/Operations/GroupBy.hs @@ -23,7 +23,7 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values groupBySingleRowWAI :: Test groupBySingleRowWAI = @@ -54,29 +54,21 @@ (print $ D.groupBy ["test0"] testData) ) -{- | The grouping invariant every backend (sequential now, parallel in Phase 2) -must satisfy: 'valueIndices' / 'offsets' / 'rowToGroup' agree, groups partition -all rows exactly once, each group is exactly the rows sharing a key tuple, and -no two groups share a key. Checked on a frame with several keys colliding under -the hash (forces the open-addressing key re-verification path). --} +-- | Every backend must partition rows into groups by key tuple. groupingInvariantHolds :: Test groupingInvariantHolds = TestCase (assertBool "grouping partitions rows by exact key tuple" (checkGrouping df)) where - -- Repeated, interleaved keys across two columns; the dense int grid is the - -- classic hash-collision stressor. + -- Repeated, interleaved keys across two columns stress hash collisions. df = - D.fromNamedColumns + D.fromColumns [ ("k1", DI.fromList (map (`mod` 5) [0 .. 199 :: Int])) , ("k2", DI.fromList (map (\i -> (i * 7) `mod` 3) [0 .. 199 :: Int])) , ("v", DI.fromList [0 .. 199 :: Int]) ] -{- | Recompute the reference grouping by the key tuple and assert the library's -'valueIndices'/'offsets'/'rowToGroup' describe the same partition. --} +-- | Recompute the reference grouping by key tuple and compare the partitions. checkGrouping :: D.DataFrame -> Bool checkGrouping df = let g = D.groupBy ["k1", "k2"] df @@ -91,13 +83,13 @@ Nothing -> error "missing column" key i = (k1 !! i, k2 !! i) -- The group slices read out of vis/os. - slices = [take (e - s) (drop s vis) | (s, e) <- zip os (tail os)] + slices = [take (e - s) (drop s vis) | (s, e) <- zip os (drop 1 os)] -- valueIndices must be a permutation of all rows. permutationOk = L.sort vis == [0 .. n - 1] -- Every slice is non-empty and internally key-constant. constKeys = all (\sl -> not (L.null sl) && allSame (map key sl)) slices -- Distinct groups have distinct keys. - groupKeys = map (key . head) slices + groupKeys = [key s | (s : _) <- slices] distinctKeys = L.length (L.nub groupKeys) == L.length groupKeys -- The number of groups equals the number of distinct key tuples. nGroupsOk = L.length slices == M.size (M.fromList [(key i, ()) | i <- [0 .. n - 1]]) diff --git a/tests/Operations/Inference.hs b/tests/Operations/Inference.hs --- a/tests/Operations/Inference.hs +++ b/tests/Operations/Inference.hs @@ -1,11 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Round-2 (S2) pins for the single-pass inference lattice and the -Int -> Double promotion path: sample classification via the WS-B byte -parsers, prefix promotion instead of full-column re-parse, and the -byte-level missing-token test in the default reader's inferred path. --} +-- | Pins for the single-pass inference lattice and Int to Double promotion. module Operations.Inference where import qualified Data.ByteString as BS @@ -29,9 +25,7 @@ assumeText :: [Maybe T.Text] -> D.ParsingAssumption assumeText = D.makeParsingAssumption "%Y-%m-%d" . V.fromList --- The candidate-mask priorities reproduce the documented fallback order: --- Bool, then Int (only when the Double mask agrees), Double, Date, Text; --- an all-null sample stays NoAssumption. +-- Candidate-mask priorities reproduce the documented fallback order. latticePriorities :: Test latticePriorities = TestCase $ do let cases = @@ -55,9 +49,7 @@ ) cases --- Pinned new behavior (WS-B parser semantics in the sample): an --- overflowing integer clears the Int candidate instead of wrapping, so --- the column classifies (and parses) as Double. +-- An overflowing integer clears the Int candidate, so the column is Double. latticeOverflowIntIsDouble :: Test latticeOverflowIntIsDouble = TestCase $ do assertEqual @@ -69,17 +61,13 @@ D.DoubleAssumption (assumeText [Just "9223372036854775808"]) --- Pinned new behavior: byte-level strip-tolerant classification accepts --- ASCII-padded numerics (the strip-equivalent unification, audit --- divergence #2), where the old Text-path sample rejected " 42" as --- Double and demoted to Text. +-- Byte-level strip-tolerant classification accepts ASCII-padded numerics. latticePaddedIntIsInt :: Test latticePaddedIntIsInt = TestCase (assertEqual "padded int" D.IntAssumption (assumeText [Just " 42 "])) --- End-to-end: an overflowing cell inside the sample now yields a Double --- column with the correctly converted value (old: silently wrapped Int). +-- An overflowing cell inside the sample yields a Double column. parseOverflowIntsAsDoubles :: Test parseOverflowIntsAsDoubles = let beforeParse = ["9223372036854775808", "1", "2"] :: [T.Text] @@ -90,8 +78,7 @@ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "overflow -> Double column" expected actual) --- Promotion: a Double cell far past the sample converts the built Int --- prefix in place; values must equal a from-scratch Double parse. +-- A Double cell far past the sample promotes the built Int prefix in place. promoteIntPrefixToDouble :: Test promoteIntPrefixToDouble = let ints = map (T.pack . show) ([1 .. 150] :: [Int]) @@ -104,8 +91,7 @@ $ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "Int prefix promoted to Double" expected actual) --- An overflowing integer past the sample triggers promotion with its --- true Double value (the old Text-path Int parse silently wrapped it). +-- An overflowing integer past the sample promotes with its true value. promoteOverflowPastSample :: Test promoteOverflowPastSample = let ints = map (T.pack . show) ([1 .. 30] :: [Int]) @@ -141,8 +127,7 @@ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "nulls survive promotion" expected actual) --- A cell that parses as neither Int nor Double demotes the whole column --- to Text with every raw cell preserved (re-extracted, not re-parsed). +-- A cell parsing as neither Int nor Double demotes the column to Text. promotionDemotesToText :: Test promotionDemotesToText = let beforeParse = map (T.pack . show) ([1 .. 30] :: [Int]) ++ ["abc"] @@ -153,8 +138,7 @@ $ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "demotes to Text, raw values kept" expected actual) --- Default (cassava) reader, inferred path: the same promotion runs over --- the retained raw bytes. +-- Default (cassava) reader, inferred path: promotion runs over raw bytes. readerPromotesIntColumn :: Test readerPromotesIntColumn = TestCase $ do let csv = @@ -170,8 +154,7 @@ Just col -> assertEqual "reader Int->Double promotion" expected col Nothing -> assertFailure "column x missing" --- Default reader: canonical missing tokens null the cell through the --- byte-level fast path, exactly as the Text-decode check did. +-- Default reader: canonical missing tokens null the cell via the fast path. readerCanonicalMissingTokens :: Test readerCanonicalMissingTokens = TestCase $ do let csv = "x,y\nNA,1\nnan,2\nNothing,3\n ,4\nNULL,5\n" @@ -185,8 +168,7 @@ Just col -> assertEqual "int column" expectedY col Nothing -> assertFailure "column y missing" --- A custom missing list replaces (not extends) the canonical one: "NA" --- must then survive as text. +-- A custom missing list replaces the canonical one, so "NA" stays text. readerCustomMissingTokens :: Test readerCustomMissingTokens = TestCase $ do let csv = "x\nfoo\nNA\n7\n" @@ -199,8 +181,7 @@ Just col -> assertEqual "custom list only" expected col Nothing -> assertFailure "column x missing" --- Default reader date columns still parse (the WS-B fast date path is --- bit-identical to readByteStringDate for %Y-%m-%d). +-- Default reader date columns still parse via the fast date path. readerDatesStillParse :: Test readerDatesStillParse = TestCase $ do let csv = "d\n2024-01-01\n2024-01-02\n2024-01-03\n" diff --git a/tests/Operations/InsertColumn.hs b/tests/Operations/InsertColumn.hs --- a/tests/Operations/InsertColumn.hs +++ b/tests/Operations/InsertColumn.hs @@ -14,7 +14,7 @@ testData :: D.DataFrame testData = - D.fromNamedColumns + D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) , ("test3", DI.fromList ([1 .. 26] :: [Int])) @@ -25,7 +25,7 @@ , ("test8", DI.fromList ['a' .. 'z']) ] --- Adding a boxed vector to an empty dataframe creates a new column boxed containing the vector elements. +-- Adding a boxed vector to an empty dataframe creates a boxed column. addBoxedColumn :: Test addBoxedColumn = TestCase @@ -53,7 +53,7 @@ ) ) --- Adding an boxed vector with an unboxable type (Int/Double) to an empty dataframe creates a new column boxed containing the vector elements. +-- Adding a boxed vector of an unboxable type creates a boxed column. addUnboxedColumn :: Test addUnboxedColumn = TestCase @@ -72,7 +72,7 @@ (DI.getColumn "new" $ D.insertColumn "new" (DI.fromList [1 :: Int, 2, 3]) D.empty) ) --- Adding a column with less values than the current DF dimensions adds column with optionals. +-- Adding a column with fewer values than the frame adds optionals. addSmallerColumnBoxed :: Test addSmallerColumnBoxed = TestCase @@ -132,7 +132,7 @@ TestCase ( assertEqual "Smaller lists should grow and contain optionals" - ( D.fromNamedColumns + ( D.fromColumns [ ( "new" , D.fromList [Just "a" :: Maybe T.Text, Just "b", Just "c", Nothing, Nothing] @@ -150,7 +150,7 @@ TestCase ( assertEqual "Smaller lists should grow and contain optionals" - ( D.fromNamedColumns + ( D.fromColumns [ ("old", D.fromList [Just 1 :: Maybe Int, Just 2, Nothing, Nothing, Nothing]) , ("new", D.fromList [Just 1 :: Maybe Int, Just 2, Just 3, Nothing, Nothing]) , ("newer", D.fromList [1 :: Int, 2, 3, 4, 5]) diff --git a/tests/Operations/InterpreterKernels.hs b/tests/Operations/InterpreterKernels.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/InterpreterKernels.hs @@ -0,0 +1,202 @@ +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE TypeApplications #-} + +-- | Kernels agree with plain Haskell row by row; floats compare bitwise. +module Operations.InterpreterKernels where + +import Control.Exception (throw) +import qualified Data.Text as T +import qualified Data.Vector as V + +import qualified DataFrame as D +import DataFrame.Expression.Operators ( + (./=.), + (.<.), + (.<=.), + (.==.), + (.>.), + (.>=.), + ) +import qualified DataFrame.Functions as F +import DataFrame.Internal.Column ( + Column, + Columnable, + TypedColumn (..), + toVector, + ) +import DataFrame.Internal.Expression (Expr (..), prettyPrint) +import DataFrame.Internal.Interpreter (interpret) + +import Assertions (BitEq, assertBitEq) +import Test.HUnit + +-- | One side of an expression: a column with known rows, or a literal. +data Operand e = Column T.Text [e] | Literal e + +operandExpr :: (Columnable e) => Operand e -> Expr e +operandExpr (Column name _) = Col name +operandExpr (Literal v) = Lit v + +-- | The operand's value in each of @n@ rows. +operandRows :: Int -> Operand e -> [e] +operandRows _ (Column _ rows) = rows +operandRows n (Literal v) = replicate n v + +operandColumns :: (Columnable e) => Operand e -> [(T.Text, Column)] +operandColumns (Column name rows) = [name D.=: rows] +operandColumns (Literal _) = [] + +-- | An operator paired with what it computes for one row. +data BinOp e r = BinOp (Expr e -> Expr e -> Expr r) (e -> e -> r) + +comparisons :: (Columnable e, Ord e) => [BinOp e Bool] +comparisons = + [ BinOp (.<.) (<) + , BinOp (.<=.) (<=) + , BinOp (.>.) (>) + , BinOp (.>=.) (>=) + , BinOp (.==.) (==) + , BinOp (./=.) (/=) + ] + +arithmetic :: (Columnable e, Num e) => [BinOp e e] +arithmetic = [BinOp (+) (+), BinOp (-) (-), BinOp (*) (*)] + +doubles :: [Double] +doubles = [0 / 0, -(1 / 0), -1.5, -0.0, 0.0, 5.0e-324, 0.5, 1, 2.5, 1.0e308, 1 / 0] + +ints :: [Int] +ints = [minBound, -3, -1, 0, 1, 7, maxBound] + +-- | The second size is above the 200k threshold where loops run in parallel. +sizes :: [Int] +sizes = [length doubles * length doubles, 200003] + +-- | Two columns holding every pair of the given values, cycled to @n@ rows. +columnPair :: Int -> [e] -> (Operand e, Operand e) +columnPair n vals = (Column "x" xs, Column "y" ys) + where + (xs, ys) = unzip (take n (cycle [(a, b) | a <- vals, b <- vals])) + +-- | Column-column, column-literal and literal-column operand pairs. +operandPairs :: Int -> [e] -> [(Operand e, Operand e)] +operandPairs n vals = (x, y) : concat [[(x, Literal t), (Literal t, x)] | t <- lits] + where + (x, y) = columnPair n vals + lits = if n > 1000 then take 3 vals else vals + +run :: forall a. (Columnable a) => [(T.Text, Column)] -> Expr a -> [a] +run columns e = case interpret @a (D.fromColumns columns) e of + Left err -> throw err + Right (TColumn col) -> V.toList (either throw id (toVector @a @V.Vector col)) + +binaryCases :: + (Columnable e, Columnable r, BitEq r, Show r) => [e] -> [BinOp e r] -> [Test] +binaryCases vals ops = + [ TestCase + (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e)) + | n <- sizes + , (l, r) <- operandPairs n vals + , let columns = operandColumns l ++ operandColumns r + , BinOp op f <- ops + , let e = op (operandExpr l) (operandExpr r) + expected = zipWith f (operandRows n l) (operandRows n r) + ] + +-- | If-then-else over a Bool column, with branches of every operand mix. +selectCases :: (Columnable e, BitEq e, Show e) => [e] -> [Test] +selectCases vals = + [ TestCase + (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e)) + | n <- sizes + , let cond = Column "c" (take n (cycle [True, False, False])) + lits = take 3 vals + , (l, r) <- + operandPairs n vals ++ zip (map Literal lits) (map Literal (drop 1 lits)) + , let columns = operandColumns cond ++ operandColumns l ++ operandColumns r + e = If (operandExpr cond) (operandExpr l) (operandExpr r) + expected = + zipWith3 + (\c a b -> if c then a else b) + (operandRows n cond) + (operandRows n l) + (operandRows n r) + ] + +caseWhenCases :: [Test] +caseWhenCases = + [ TestCase + (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e)) + | n <- sizes + , let x = fst (columnPair n ints) + columns = operandColumns x + xe = operandExpr x + e = + F.caseWhen (xe .>. Lit 5) (Lit 1) $ + F.caseWhen (xe .>. Lit 0) (Lit 2) $ + F.orElse (Lit (3 :: Int)) + expected = [if v > 5 then 1 else if v > 0 then 2 else 3 | v <- operandRows n x] + ] + +toDoubleCases :: [Test] +toDoubleCases = + [ TestCase + ( assertBitEq + (prettyPrint e ++ " n=" ++ show n) + (map fromIntegral (operandRows n x)) + (run (operandColumns x) e) + ) + | n <- sizes + , let x = fst (columnPair n ints) + e = F.toDouble (operandExpr x) + ] + +mergeAggCases :: [Test] +mergeAggCases = concatMap sizeCases sizes + where + sizeCases n = + [ TestCase $ + assertEqual + ("count over " ++ show n ++ " rows") + [n] + (take 1 (run (operandColumns xsI) (F.count (F.col @Int "x")))) + , TestCase $ + assertEqual + ("countDistinct of an expression over " ++ show n ++ " rows") + [length (takeWhile (< n) [0 .. 6])] + (take 1 (run (operandColumns xsI) (F.countDistinct (F.col @Int "x" + 1)))) + , TestCase $ + assertBool + ("mean of ints over " ++ show n ++ " rows") + ( all + (approx (sumD / fromIntegral n)) + (run (operandColumns xsI) (F.mean (F.col @Int "x"))) + ) + , TestCase $ + assertEqual + ("mean of doubles over " ++ show n ++ " rows") + [sum (take n smallDoubles) / fromIntegral n] + (take 1 (run (operandColumns xsD) (F.mean (F.col @Double "x")))) + ] + where + -- Seven distinct ints with a large spread; the sum fits in a Double. + xs = take n (cycle [3, -1, 0, 1, 7, 1000000007, -999999989 :: Int]) + xsI = Column "x" xs + xsD = Column "x" (take n smallDoubles) + sumD = sum (map fromIntegral xs) :: Double + smallDoubles = cycle [0.5, 1, 2.5, -3, 0.25] + approx a b = abs (a - b) <= 1.0e-9 * max 1 (abs a) + +tests :: [Test] +tests = + mergeAggCases + ++ binaryCases doubles comparisons + ++ binaryCases ints comparisons + ++ binaryCases doubles arithmetic + ++ binaryCases ints arithmetic + ++ selectCases doubles + ++ selectCases ints + ++ caseWhenCases + ++ toDoubleCases diff --git a/tests/Operations/Join.hs b/tests/Operations/Join.hs --- a/tests/Operations/Join.hs +++ b/tests/Operations/Join.hs @@ -16,14 +16,14 @@ df1 :: D.DataFrame df1 = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) ] df2 :: D.DataFrame df2 = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) ] @@ -48,7 +48,7 @@ TestCase ( assertEqual "Test inner join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) @@ -62,7 +62,7 @@ TestCase ( assertEqual "Test left join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"]) @@ -76,7 +76,7 @@ TestCase ( assertEqual "Test right join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList [Just "A0" :: Maybe Text, Just "A1", Just "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) @@ -96,7 +96,7 @@ TestCase ( assertEqual "Test typed inner join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) @@ -110,7 +110,7 @@ TestCase ( assertEqual "Test typed left join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"]) @@ -122,7 +122,7 @@ -- A right-hand frame whose payload column is already optional. dfOptional :: D.DataFrame dfOptional = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1"]) , ("C", D.fromList [Just 10 :: Maybe Int, Just 11]) ] @@ -131,16 +131,13 @@ DT.TypedDataFrame ['("key", Text), '("C", Maybe Int)] tdfOptional = either (error . show) id (DT.freezeWithError dfOptional) -{- | A left join over an already-optional column must not nest the Maybe: the -explicit @Maybe Int@ result schema below only type-checks because 'WrapMaybe' -flattens @Maybe (Maybe Int)@ to @Maybe Int@, matching the runtime column. --} +-- | A left join over an already-optional column must not nest the Maybe. testLeftJoinTypedOptional :: Test testLeftJoinTypedOptional = TestCase ( assertEqual "Typed left join keeps an already-optional column single-Maybe" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) , @@ -163,7 +160,7 @@ TestCase ( assertEqual "Test typed right join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList [Just "A0" :: Maybe Text, Just "A1", Just "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) @@ -195,7 +192,7 @@ TestCase ( assertEqual "Test full outer join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ( "Name" , D.fromList ["James" :: Text, "Kelly", "Mike", "Sally"] @@ -224,7 +221,7 @@ TestCase ( assertEqual "Full outer join on an unboxed (Int) key column coalesces correctly" - ( D.fromNamedColumns + ( D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 3, 4]) , ( "amount" @@ -244,21 +241,21 @@ ordersDf :: D.DataFrame ordersDf = - D.fromNamedColumns + D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 4]) , ("amount", D.fromList [250.0 :: Double, 80.0, 125.0]) ] customersDf :: D.DataFrame customersDf = - D.fromNamedColumns + D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 3]) , ("name", D.fromList ["Ada" :: Text, "Lin", "Grace"]) ] dfL :: D.DataFrame dfL = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("X", D.fromList ["LX0" :: Text, "LX1", "LX2"]) , ("Lonly", D.fromList ["L0" :: Text, "L1", "L2"]) @@ -266,7 +263,7 @@ dfR :: D.DataFrame dfR = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K3"]) , ("X", D.fromList ["RX0" :: Text, "RX1", "RX3"]) , ("Ronly", D.fromList [10 :: Int, 11, 13]) @@ -277,7 +274,7 @@ TestCase ( assertEqual "Test inner join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1"]) , ("X", D.fromList [These "LX0" "RX0" :: These Text Text, These "LX1" "RX1"]) , ("Lonly", D.fromList ["L0" :: Text, "L1"]) @@ -292,7 +289,7 @@ TestCase ( assertEqual "Test left join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ( "X" @@ -310,7 +307,7 @@ TestCase ( assertEqual "Test right join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K3"]) , ( "X" @@ -328,7 +325,7 @@ TestCase ( assertEqual "Test right join with single key" - ( D.fromNamedColumns + ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3"]) , ( "X" @@ -390,7 +387,7 @@ testInnerJoinMissingKeysSuggestion = TestCase $ let typoDf = - D.fromNamedColumns + D.fromColumns [ ("hello", D.fromList ["H" :: Text]) , ("world", D.fromList ["W" :: Text]) ] @@ -402,14 +399,14 @@ -- Empty DataFrame fixtures: same schema as df1/df2 but zero rows. emptyDf1 :: D.DataFrame emptyDf1 = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ([] :: [Text])) , ("A", D.fromList ([] :: [Text])) ] emptyDf2 :: D.DataFrame emptyDf2 = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ([] :: [Text])) , ("B", D.fromList ([] :: [Text])) ] @@ -451,19 +448,16 @@ ) -- Many-to-many: duplicate keys on both sides produce the cross-product. --- manyLeft: K0->A0, K1->A1, K1->A2 --- manyRight: K0->B0, K1->B1, K1->B2 --- Expected inner join: 1 K0 pair + 4 K1 pairs = 5 rows manyLeft :: D.DataFrame manyLeft = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K1"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) ] manyRight :: D.DataFrame manyRight = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K1"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) ] @@ -487,17 +481,16 @@ ) -- Stress the open-addressing hash index with many distinct integer keys. --- Left keys 0..9999, right keys 5000..14999 (overlap 5000..9999). bigLeft :: D.DataFrame bigLeft = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList [0 .. 9999 :: Int]) , ("la", D.fromList [i * 2 | i <- [0 .. 9999 :: Int]]) ] bigRight :: D.DataFrame bigRight = - D.fromNamedColumns + D.fromColumns [ ("key", D.fromList [5000 .. 14999 :: Int]) , ("rb", D.fromList [i * 3 | i <- [5000 .. 14999 :: Int]]) ] diff --git a/tests/Operations/Merge.hs b/tests/Operations/Merge.hs --- a/tests/Operations/Merge.hs +++ b/tests/Operations/Merge.hs @@ -9,32 +9,32 @@ df1 :: D.DataFrame df1 = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList [1 :: Int, 2]) , ("B", DI.fromList ['a', 'b']) ] df2 :: D.DataFrame df2 = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList [3 :: Int]) , ("B", DI.fromList ['c']) ] dfOnlyA :: D.DataFrame -dfOnlyA = D.fromNamedColumns [("A", DI.fromList [1 :: Int, 2])] +dfOnlyA = D.fromColumns [("A", DI.fromList [1 :: Int, 2])] dfOnlyB :: D.DataFrame -dfOnlyB = D.fromNamedColumns [("B", DI.fromList ['c', 'd'])] +dfOnlyB = D.fromColumns [("B", DI.fromList ['c', 'd'])] dfSide1 :: D.DataFrame dfSide1 = - D.fromNamedColumns + D.fromColumns [("L", DI.fromList [10 :: Int, 20])] dfSide2 :: D.DataFrame dfSide2 = - D.fromNamedColumns + D.fromColumns [("R", DI.fromList ["x" :: String, "y"])] mergeVerticalWAI :: Test @@ -42,7 +42,7 @@ TestCase ( assertEqual "Vertical merge concatenates matching columns" - ( D.fromNamedColumns + ( D.fromColumns [ ("A", DI.fromList [1 :: Int, 2, 3]) , ("B", DI.fromList ['a', 'b', 'c']) ] @@ -55,7 +55,7 @@ TestCase ( assertEqual "Disjoint schemas produce optional padding" - ( D.fromNamedColumns + ( D.fromColumns [ ( "A" , DI.fromList @@ -102,7 +102,7 @@ TestCase ( assertEqual "Horizontal merge combines columns side by side" - ( D.fromNamedColumns + ( D.fromColumns [ ("L", DI.fromList [10 :: Int, 20]) , ("R", DI.fromList ["x" :: String, "y"]) ] @@ -131,14 +131,14 @@ mergeVerticalDifferentTypesSameColumnName :: Test mergeVerticalDifferentTypesSameColumnName = let - dfIntA = D.fromNamedColumns [("A", DI.fromList [1 :: Int, 2])] + dfIntA = D.fromColumns [("A", DI.fromList [1 :: Int, 2])] - dfTextA = D.fromNamedColumns [("A", DI.fromList ["x" :: String, "y"])] + dfTextA = D.fromColumns [("A", DI.fromList ["x" :: String, "y"])] in TestCase ( assertEqual "Merging columns with same name but different types wraps the values in Either" - ( D.fromNamedColumns + ( D.fromColumns [ ("A", DI.fromList [Left (1 :: Int), Left 2, Right ("x" :: String), Right "y"]) ] ) diff --git a/tests/Operations/MonomorphicLoops.hs b/tests/Operations/MonomorphicLoops.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/MonomorphicLoops.hs @@ -0,0 +1,324 @@ +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE TypeApplications #-} + +-- | Per-type column loops checked against list references and null handling. +module Operations.MonomorphicLoops where + +import Control.Exception (throw) +import qualified Data.Text as T +import qualified Data.Vector as V +import qualified Data.Vector.Unboxed as VU +import Data.Word (Word64) +import GHC.Float (castDoubleToWord64, castFloatToWord32) + +import DataFrame.Internal.Column ( + Column, + Columnable, + findIndices, + foldLinearGroups, + foldl1Column, + foldl1DirectGroups, + foldlColumn, + fromList, + fromUnboxedVector, + headColumn, + ifoldrColumn, + imapColumn, + mapColumn, + toVector, + zipColumns, + zipWithColumns, + ) + +import Test.HUnit + +sizes :: [Int] +sizes = [37, 200003] + +doublesOf :: Int -> [Double] +doublesOf n = take n (cycle [0 / 0, -(1 / 0), -1.5, -0.0, 0.0, 5.0e-324, 0.5, 2.5, 1 / 0]) + +intsOf :: Int -> [Int] +intsOf n = take n (cycle [minBound, -3, 0, 1, 7, maxBound]) + +boolsOf :: Int -> [Bool] +boolsOf n = take n (cycle [True, False, False]) + +col :: (Columnable a, VU.Unbox a) => [a] -> Column +col = fromUnboxedVector . VU.fromList + +out :: forall a e. (Columnable a) => Either e Column -> [a] +out (Left _) = error "column op failed" +out (Right c) = V.toList (either throw id (toVector @a @V.Vector c)) + +-- | Element results compare by bits for floating types, by '==' otherwise. +class Same a where + same :: [a] -> [a] -> Bool + +instance Same Double where + same a b = map castDoubleToWord64 a == map castDoubleToWord64 b +instance Same Int where + same = (==) +instance Same Bool where + same = (==) +instance Same Float where + same a b = map castFloatToWord32 a == map castFloatToWord32 b + +bits :: [Double] -> [Word64] +bits = map castDoubleToWord64 + +-- | map, imap and zip of one (element, result) pairing. +pairCases :: + forall a c. + (Columnable a, VU.Unbox a, Columnable c, Same c) => + String -> (Int -> [a]) -> (a -> c) -> (Int -> a -> c) -> (a -> a -> c) -> [Test] +pairCases label gen f fi f2 = + [ TestList + [ TestCase + ( assertBool + (label ++ " map n=" ++ show n) + (same (out @c (mapColumn f (col xs))) (map f xs)) + ) + , TestCase + ( assertBool + (label ++ " imap n=" ++ show n) + (same (out @c (imapColumn fi (col xs))) (zipWith fi [0 ..] xs)) + ) + , TestCase + ( assertBool + (label ++ " zip n=" ++ show n) + (same (out @c (zipWithColumns f2 (col xs) (col ys))) (zipWith f2 xs ys)) + ) + ] + | n <- sizes + , let xs = gen n + ys = reverse xs + ] + +foldCases :: [Test] +foldCases = + concat + [ [ TestCase + ( assertEqual + ("foldl DD n=" ++ show n) + (bits [foldl (+) 0.25 ds]) + (bits [fold (+) 0.25 ds]) + ) + , TestCase + ( assertEqual + ("foldl DI n=" ++ show n) + (foldl (\k d -> k + fromEnum (d > 0)) 0 ds) + (fold (\k d -> k + fromEnum (d > 0)) 0 ds) + ) + , TestCase + ( assertEqual + ("foldl ID n=" ++ show n) + (bits [foldl (\a i -> a + fromIntegral i) 0 is]) + (bits [fold (\a i -> a + fromIntegral i) (0 :: Double) is]) + ) + , TestCase + (assertEqual ("foldl II n=" ++ show n) (foldl (+) 1 is) (fold (+) 1 is)) + , TestCase + ( assertEqual + ("foldl1 D n=" ++ show n) + (bits [foldl1 (-) ds]) + (bits [fold1 (-) ds]) + ) + , TestCase (assertEqual ("foldl1 I n=" ++ show n) (maximum is) (fold1 max is)) + , TestCase + ( assertEqual + ("foldl Float n=" ++ show n) + (castFloatToWord32 (sum fs)) + (castFloatToWord32 (fold (+) (0 :: Float) fs)) + ) + ] + | n <- sizes + , let ds = doublesOf n + is = intsOf n + fs = map realToFrac ds :: [Float] + ] + where + fold :: + (Columnable a, VU.Unbox a, Columnable b) => (b -> a -> b) -> b -> [a] -> b + fold f z xs = either throw id (foldlColumn f z (col xs)) + fold1 :: (Columnable a, VU.Unbox a) => (a -> a -> a) -> [a] -> a + fold1 f xs = either throw id (foldl1Column f (col xs)) + +-- | Mapping at the element type over a nullable column keeps its bitmap. +nullableCases :: [Test] +nullableCases = + [ TestCase + ( assertEqual + ("nullable map n=" ++ show n) + (map (fmap (castDoubleToWord64 . (* 2))) xs) + ( map + (fmap castDoubleToWord64) + (out @(Maybe Double) (mapColumn @Double (* 2) (fromList xs))) + ) + ) + | n <- sizes + , let xs = + take n (cycle [Just 1.5, Nothing, Just (-0.0), Just (0 / 0)]) :: [Maybe Double] + ] + +-- Functions over a column with nulls: f never runs on a null row. +nullSemantics :: [Test] +nullSemantics = + [ eq + "map skips null rows" + [Just 25, Nothing, Just 50] + (outM @Int (mapColumn @Int (100 `div`) ints3)) + , eq + "map from boxed nullable" + [Just 2, Nothing, Just 0] + (outM @Int (mapColumn @T.Text T.length texts3)) + , eq + "map to Maybe" + [Just 4, Nothing, Nothing] + (outM @Int (mapColumn @Int keepBig ints3)) + , eq + "imap skips null rows" + [Just 4, Nothing, Just 4] + (outM @Int (imapColumn @Int (flip (+)) ints3)) + , eq + "zip: null if either side is" + [Just 5, Nothing, Nothing] + (outM @Int (zipWithColumns @Int @Int (\a b -> a + b `div` b) ints3 other3)) + , eq + "zip to Maybe" + [Nothing, Nothing, Just 3] + ( outM @Int + ( zipWithColumns @Int @Int + (\a b -> if a < b then Just (b - a) else Nothing) + ints3 + other3' + ) + ) + , eq "foldl skips null rows" (Just 6) (ok $ foldlColumn @Int (+) 0 ints3) + , eq "foldl1 skips null rows" (Just 4) (ok $ foldl1Column @Int max ints3) + , TestCase + ( assertBool + "foldl1 over only nulls fails" + ( either + (const True) + (const False) + (foldl1Column @Int (+) (fromList [Nothing, Nothing :: Maybe Int])) + ) + ) + , eq + "ifoldr skips null rows" + (Just [0, 2]) + (ok $ ifoldrColumn @Int (\i _ r -> i : r) [] ints3) + , eq + "findIndices skips null rows" + (Just (VU.fromList [0])) + (ok $ findIndices @Int (== 4) (fromList [Just 4, Nothing, Just (1 :: Int)])) + , eq + "findIndices never matches a null" + (Just VU.empty) + (ok $ findIndices @Int (== 0) (fromList [Just 3, Nothing :: Maybe Int])) + , eq + "head is the first valid row" + (Just 3) + (ok $ headColumn @Int (fromList [Nothing, Just (3 :: Int)])) + , eq + "zipColumns keeps nulls" + [Just (4, 1), Nothing, Nothing] + (outM @(Int, Int) (Right (zipColumns ints3 other3))) + , eq + "foldLinearGroups skips null rows" + [1, 2] + ( out @Int + ( foldLinearGroups @Int + (+) + 0 + (fromList [Just 1, Nothing, Just 2, Nothing :: Maybe Int]) + (VU.fromList [0, 0, 1, 1]) + 2 + ) + ) + , eq + "foldl1DirectGroups: all-null group is null" + [Nothing, Just 7] + ( outM @Int + ( foldl1DirectGroups @Int + (+) + (fromList [Nothing, Nothing, Just 3, Just (4 :: Int)]) + (VU.fromList [0 .. 3]) + (VU.fromList [0, 2, 4]) + ) + ) + , TestCase + ( assertBool + "toVector refuses an element read with nulls" + (either (const True) (const False) (toVector @Int @V.Vector ints3)) + ) + , eq + "toVector allows a bitmap with no nulls" + (Just [1, 2]) + (ok $ V.toList <$> toVector @Int @V.Vector (fromList [Just 1, Just (2 :: Int)])) + ] + where + ok :: Either e x -> Maybe x + ok = either (const Nothing) Just + eq :: (Eq x, Show x) => String -> x -> x -> Test + eq label expected actual = TestCase (assertEqual label expected actual) + ints3 = fromList [Just 4, Nothing, Just (2 :: Int)] + other3 = fromList [Just 1, Just 0, Nothing :: Maybe Int] + other3' = fromList [Just 1, Just 0, Just (5 :: Int)] + texts3 = fromList [Just ("ab" :: T.Text), Nothing, Just ""] + keepBig x = if x > 3 then Just x else Nothing + +outM :: forall a e. (Columnable a) => Either e Column -> [Maybe a] +outM (Left _) = error "column op failed" +outM (Right c) = V.toList (either throw id (toVector @(Maybe a) @V.Vector c)) + +tests :: [Test] +tests = + concat + [ nullableCases + , nullSemantics + , pairCases @Double @Double + "DD" + doublesOf + (* 1.5) + (\i d -> d + fromIntegral i) + (-) + , pairCases @Double @Int + "DI" + doublesOf + (fromEnum . (> 0)) + (\i d -> i + fromEnum (d < 0)) + (\a b -> fromEnum (a <= b)) + , pairCases @Double @Bool "DB" doublesOf (<= 0.5) (\i d -> even i || d > 0) (<) + , pairCases @Int @Double + "ID" + intsOf + fromIntegral + (\i x -> fromIntegral (x + i)) + (\a b -> fromIntegral a - fromIntegral b) + , pairCases @Int @Int "II" intsOf (* 3) (+) (-) + , pairCases @Int @Bool "IB" intsOf odd (flip (>)) (==) + , pairCases @Bool @Double + "BD" + boolsOf + (\b -> if b then 2.5 else 0 / 0) + (\i b -> if b then fromIntegral i else -0.0) + (\a b -> if a && b then 1 else -1) + , pairCases @Bool @Int + "BI" + boolsOf + fromEnum + (\i b -> if b then i else -i) + (\a b -> fromEnum a + fromEnum b) + , pairCases @Bool @Bool "BB" boolsOf not (\i b -> b /= even i) (/=) + , pairCases @Float @Float + "Float fallback" + (map realToFrac . doublesOf) + (* 2) + (\i x -> x + fromIntegral i) + (+) + , foldCases + ] diff --git a/tests/Operations/Nullable.hs b/tests/Operations/Nullable.hs --- a/tests/Operations/Nullable.hs +++ b/tests/Operations/Nullable.hs @@ -20,14 +20,10 @@ import Test.HUnit --- --------------------------------------------------------------------------- --- Untyped Expr layer tests --- --------------------------------------------------------------------------- - -- DataFrame with one plain Int column and one Maybe Int column testData :: D.DataFrame testData = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1, 2, 3] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 10, Nothing, Just 30 :: Maybe Int])) ] @@ -141,10 +137,6 @@ ) ) --- --------------------------------------------------------------------------- --- nullLift / nullLift2 — untyped layer --- --------------------------------------------------------------------------- - -- | nullLift negate on Maybe Int propagates Nothing nullLiftMaybeInt :: Test nullLiftMaybeInt = @@ -222,14 +214,10 @@ ) ) --- --------------------------------------------------------------------------- --- Cross-type arithmetic (Int × Double promotion) --- --------------------------------------------------------------------------- - -- DataFrame with Int, Maybe Int, Double, Maybe Double columns crossData :: D.DataFrame crossData = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1, 2, 3] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 10, Nothing, Just 30 :: Maybe Int])) , ("d", DI.fromList ([1.5, 2.5, 3.5] :: [Double])) @@ -451,7 +439,7 @@ -- Division tests use clean-dividing data: x=[2,4,6], d=[1,2,3] divData :: D.DataFrame divData = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([2, 4, 6] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 4, Nothing, Just 6 :: Maybe Int])) , ("d", DI.fromList ([1.0, 2.0, 3.0] :: [Double])) @@ -527,10 +515,6 @@ ) ) --- --------------------------------------------------------------------------- --- Typed TExpr layer — cross-type arithmetic --- --------------------------------------------------------------------------- - type CrossSchema = '[ '("x", Int) , '("y", Maybe Int) @@ -615,10 +599,6 @@ ) ) --- --------------------------------------------------------------------------- --- Typed TExpr layer tests --- --------------------------------------------------------------------------- - type TestSchema = '[ '("x", Int), '("y", Maybe Int)] typedTestData :: DT.TypedDataFrame TestSchema @@ -725,10 +705,6 @@ ) ) --- --------------------------------------------------------------------------- --- Grouped aggregation over nullable columns --- --------------------------------------------------------------------------- - -- | Interleaved, so grouping's row permutation is not the identity. interleavedKeys :: [T.Text] interleavedKeys = ["a", "b", "a", "b", "a", "b", "a", "b"] @@ -740,7 +716,7 @@ nullsInOneGroup :: D.DataFrame nullsInOneGroup = - D.fromNamedColumns + D.fromColumns [ ("k", DI.fromList interleavedKeys) , ("v", DI.fromVector (V.fromList nullsInOneGroupValues)) ] @@ -752,7 +728,7 @@ nullsInBothGroups :: D.DataFrame nullsInBothGroups = - D.fromNamedColumns + D.fromColumns [ ("k", DI.fromList interleavedKeys) , ("v", DI.fromVector (V.fromList nullsInBothGroupsValues)) ] @@ -766,7 +742,7 @@ expectedGroupSums :: [Double] -> D.DataFrame expectedGroupSums sums = - D.fromNamedColumns + D.fromColumns [ ("k", DI.fromList (["a", "b"] :: [T.Text])) , ("s", DI.fromList sums) ] diff --git a/tests/Operations/NullableHashing.hs b/tests/Operations/NullableHashing.hs --- a/tests/Operations/NullableHashing.hs +++ b/tests/Operations/NullableHashing.hs @@ -1,14 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | -Regression tests for null-aware row hashing and row extraction. - -`Nothing` in a nullable unboxed column must be a distinct value from any -`Just x` (so @distinct@/@groupBy@/joins do not merge them), and `toRowList` -must round-trip nulls faithfully. These pin the bug surfaced by the -category-theory set-algebra laws, where @Nothing@ was stored as an -uninitialised int and collided with @Just 0@. --} +-- | Regression tests for null-aware row hashing and row extraction. module Operations.NullableHashing where import qualified DataFrame as D @@ -18,7 +10,7 @@ import Test.HUnit maybeIntCol :: [Maybe Int] -> D.DataFrame -maybeIntCol xs = D.fromNamedColumns [("k", DI.fromList xs)] +maybeIntCol xs = D.fromColumns [("k", DI.fromList xs)] -- | distinct must keep @Nothing@ and @Just 0@ as two distinct rows. distinctSeparatesNullFromZero :: Test @@ -40,7 +32,7 @@ ( fst ( D.dimensions ( D.distinct - ( D.fromNamedColumns + ( D.fromColumns [ ("k", DI.fromList [Just (0 :: Int), Nothing]) , ("v", DI.fromList ['a', 'a']) ] @@ -81,17 +73,13 @@ rebuilt = D.fromRows ["k"] (map (map snd) (toRowList df)) in TestCase (assertEqual "toRowList round-trip preserves nulls" df rebuilt) -{- | Hash robustness: @distinct@ must preserve every row of a dense grid of -small integers across several columns. A weak per-step hash collides badly on -adjacent integers and merges distinct rows; grouping trusts hash equality, so -this guards that the hash spreads such keys. --} +-- | distinct must preserve every row of a dense grid of small integers. denseIntGridNoCollisions :: Test denseIntGridNoCollisions = let d = [-4 .. 4 :: Int] rows = [(a, b, c) | a <- d, b <- d, c <- d] df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList (map (\(a, _, _) -> a) rows)) , ("b", DI.fromList (map (\(_, b, _) -> b) rows)) , ("c", DI.fromList (map (\(_, _, c) -> c) rows)) @@ -103,16 +91,13 @@ (fst (D.dimensions (D.distinct df))) ) -{- | Join-hash robustness: a self inner-join on a dense grid of unique integer -keys must return exactly one match per row. Joins match purely by hash, so a -weak hash that collides distinct keys would emit spurious cross-matches. --} +-- | A self inner-join on a dense unique-key grid returns one match per row. joinDenseGridNoCollisions :: Test joinDenseGridNoCollisions = let d = [-4 .. 4 :: Int] rows = [(a, b) | a <- d, b <- d] df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList (map fst rows)) , ("b", DI.fromList (map snd rows)) ] @@ -129,7 +114,7 @@ let d = Nothing : map Just [-3 .. 3 :: Int] rows = [(a, b) | a <- d, b <- d] df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList (map fst rows)) , ("b", DI.fromList (map snd rows)) ] diff --git a/tests/Operations/ParallelGroupBy.hs b/tests/Operations/ParallelGroupBy.hs --- a/tests/Operations/ParallelGroupBy.hs +++ b/tests/Operations/ParallelGroupBy.hs @@ -1,23 +1,22 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The parallel-grouping correctness gate. Every case asserts that the -parallel partitioned grouping ('groupByPar') produces a 'Grouped' value -bit-for-bit identical to the sequential reference ('groupBySeq') — -'valueIndices', 'offsets' and 'rowToGroup' all equal — and that aggregating -through both yields the same result. Covered: single/multi key, Int/Text/Double -keys, heavy hash collisions, null patterns, and row counts straddling the -parallel threshold. --} +-- | Parallel grouping must be bit-for-bit identical to the sequential one. module Operations.ParallelGroupBy where +import qualified Data.ByteString as BS import qualified Data.Map.Strict as M import qualified Data.Text as T +import qualified Data.Text.Array as A +import qualified Data.Text.Encoding as TE +import Data.Text.Internal (Text (Text)) import qualified Data.Vector.Unboxed as VU import qualified DataFrame as D +import DataFrame.Expression.Operators ((./=.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI +import DataFrame.Internal.Data.PackedText (mkPackedContiguous) import qualified DataFrame.Internal.DataFrame as DD import DataFrame.Internal.Grouping (groupByPar, groupBySeq) @@ -46,7 +45,7 @@ -- | Build a frame of @n@ rows with several key columns and a value column. caseFrame :: Int -> DD.DataFrame caseFrame n = - D.fromNamedColumns + D.fromColumns [ mkInt "ki" [i `mod` 997 | i <- [0 .. n - 1]] , mkText "kt" [T.pack ("g" ++ show (i `mod` 503)) | i <- [0 .. n - 1]] , mkDouble "kd" [fromIntegral (i `mod` 311) / 7 | i <- [0 .. n - 1]] @@ -60,7 +59,7 @@ -- | Collision-heavy small grid (forces hash-collision key re-verification). collisionFrame :: DD.DataFrame collisionFrame = - D.fromNamedColumns + D.fromColumns [ mkInt "k1" (map (`mod` 5) [0 .. 999]) , mkInt "k2" (map (\i -> (i * 7) `mod` 3) [0 .. 999]) , mkInt "v" [0 .. 999] @@ -77,9 +76,7 @@ , ["ki", "kt", "kd", "kn"] ] -{- | Sizes straddling the 200k parallel threshold (the parallel path only kicks -in for the larger ones at runtime, but 'groupByPar' is exercised on all). --} +-- | Sizes straddling the 200k parallel threshold. sizes :: [Int] sizes = [1, 50, 4999, 200001, 350000] @@ -121,15 +118,10 @@ (groupByPar ["k1", "k2"] collisionFrame) ) -{- | The Q9 regression family: six derived/sum aggregations that together form -the moment sufficient statistics (count, Sx, Sy, Sxx, Syy, Sxy). The fused -'momentScatter' path must produce the same numbers as a deterministic oracle -computed from the raw data, at both -N1 and -N8 (parallel==sequential). The -input has columns the derive step turns into v1*v1, v1*v2, v2*v2. --} +-- | Frame for the moment-statistics aggregations checked against an oracle. momentFrame :: Int -> DD.DataFrame momentFrame n = - D.fromNamedColumns + D.fromColumns [ mkInt "id2" [i `mod` 17 | i <- [0 .. n - 1]] , mkInt "id4" [(i * 3) `mod` 11 | i <- [0 .. n - 1]] , mkInt "v1" [(i * 7) `mod` 100 | i <- [0 .. n - 1]] @@ -165,11 +157,7 @@ (momentResult groupBySeq n) (momentResult groupByPar n) -{- | Independent oracle: recompute the six moment sums per group with plain -@Data.Map@ folds in original-row order and check the aggregated frame matches. -This pins the fused kernel against a non-scatter reference (counts exact, sums -fold in the same row order so they are byte-identical). --} +-- | Oracle recomputing the six moment sums per group with plain Map folds. momentOracle :: Int -> Test momentOracle n = TestCase $ do @@ -183,20 +171,12 @@ Nothing -> -1 assertEqual ("oracle group-row count n=" ++ show n) refN outN -{- | The low-cardinality DIRECT-INDEXED grouping fast path -('DataFrame.Internal.Grouping.Direct') fires from 'D.groupBy' on a single clean -small-range Int key. It emits groups in ascending key-value order rather than the -hash path's order, so we cannot compare index structures directly; instead we -assert it produces the SAME per-key aggregate values as the hash 'groupBySeq' -reference. The per-group v-sum, count, min and max are exact integers, so they -must match the hash path's results key-for-key (independent of group order). --} +-- | Direct-indexed grouping must give the same per-key aggregates as hashing. directGroupAggMatches :: Int -> Test directGroupAggMatches n = TestCase $ let df = caseFrame n v = F.col @Int "v" - ki = F.col @Int "ki" aggs = [ "vsum" F..= F.sum v , "vcount" F..= F.count v @@ -211,9 +191,7 @@ (keyedAgg hashRes) (keyedAgg directRes) -{- | Read an aggregated frame keyed on @ki@ into a 'M.Map' from key to the four -aggregate values, so two frames can be compared independent of group order. --} +-- | Read an aggregated frame into a Map keyed by group, ignoring group order. keyedAgg :: DD.DataFrame -> M.Map Int (Int, Int, Int, Int) keyedAgg res = M.fromList @@ -225,12 +203,39 @@ zip4 (a : as) (b : bs) (c : cs) (d : ds) = (a, b, c, d) : zip4 as bs cs ds zip4 _ _ _ _ = [] +-- | A packed UTF-8 column, the layout the Parquet reader produces. +packedCol :: [T.Text] -> DI.Column +packedCol ts = + let bytes = BS.concat (map TE.encodeUtf8 ts) + offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts) + Text arr _ _ = TE.decodeUtf8 bytes + in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs)) + +-- | Skewed packed-text keys with a few hot keys, grouped in parallel. +skewedTextParity :: Bool -> Test +skewedTextParity filtered = TestCase $ do + let n = 300000 + key i + | i `mod` 3 == 0 = "" + | i `mod` 5 == 0 = "hot" + | i `mod` 7 == 0 = "warm" + | otherwise = T.pack ("k" ++ show ((i * 7919) `mod` 40000)) + df0 = + D.fromColumns [("k", packedCol (map key [0 .. n - 1])), mkInt "v" [0 .. n - 1]] + df = if filtered then D.filterWhere (F.col @T.Text "k" ./=. "") df0 else df0 + assertBool + "skewed packed text: par == seq" + (sameGrouping (groupByPar ["k"] df) (groupBySeq ["k"] df)) + tests :: [Test] tests = - [ TestLabel ("parity " ++ show n ++ " " ++ show keys) (parityFor n keys) - | n <- sizes - , keys <- keySets + [ TestLabel ("skewedTextParity " ++ show f) (skewedTextParity f) + | f <- [False, True] ] + ++ [ TestLabel ("parity " ++ show n ++ " " ++ show keys) (parityFor n keys) + | n <- sizes + , keys <- keySets + ] ++ [TestLabel ("aggParity " ++ show n) (aggParityFor n) | n <- [50, 200001]] ++ [TestLabel "collisionParity" collisionParity] ++ [TestLabel ("momentParity " ++ show n) (momentParity n) | n <- [3, 50, 200001]] diff --git a/tests/Operations/ParallelJoin.hs b/tests/Operations/ParallelJoin.hs --- a/tests/Operations/ParallelJoin.hs +++ b/tests/Operations/ParallelJoin.hs @@ -1,13 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | The parallel-join correctness gate. Each case asserts that the parallel -chunked-probe kernels ('parInnerKernel' \/ 'parLeftKernel') produce index -vectors /bit-for-bit identical/ to the sequential reference kernels, and that -the assembled join through the public 'innerJoin' \/ 'leftJoin' matches a -sequentially-forced reference. Covered: int\/text keys, many-to-many duplicate -keys, unmatched left rows, null patterns, and row counts straddling the -parallel threshold. --} +-- | Parallel join kernels must match the sequential reference bit for bit. module Operations.ParallelJoin where import qualified Data.Text as T @@ -31,9 +24,7 @@ mkHashes :: [Int] -> VU.Vector Int mkHashes = VU.fromList -{- | Synthetic hash vectors. Many collisions (mod), duplicates on both sides, -and sizes that exceed the parallel threshold so the parallel path engages. --} +-- | Synthetic hash vectors with collisions, duplicates and large sizes. probeBuildPairs :: [(String, HashPair)] probeBuildPairs = [ ("tiny", (mkHashes [1, 2, 3], mkHashes [2, 3, 4])) @@ -52,10 +43,7 @@ , mkHashes [i * 3 | i <- [0 .. 120000 :: Int]] ) ) - , -- Small-build (cache-resident) build side probed by a very large probe: - -- the parallel small-build-probe lever (medium-factor regime). The probe - -- exceeds parProbeThreshold so the parallel path engages with a tiny - -- read-only shared index. Parity here is the new correctness gate. + , -- Small-build side probed by a very large probe; parity is the gate. ( "small-build-big-probe" , @@ -86,22 +74,19 @@ -- | Build a frame large enough to cross the parallel threshold. ordersFrame :: Int -> D.DataFrame ordersFrame n = - D.fromNamedColumns + D.fromColumns [ ("cid", D.fromList [i `mod` 7000 | i <- [0 .. n - 1]]) , ("amount", D.fromList [fromIntegral i * 1.5 :: Double | i <- [0 .. n - 1]]) ] custFrame :: Int -> D.DataFrame custFrame n = - D.fromNamedColumns + D.fromColumns [ ("cid", D.fromList [0 .. n - 1]) , ("region", D.fromList [T.pack ('r' : show (i `mod` 5)) | i <- [0 .. n - 1]]) ] -{- | End-to-end: the assembled inner/left join over a frame that exceeds the -parallel threshold equals the same join sorted (order-independent value check), -and the result row count is stable. --} +-- | End to end: a large join equals the sorted join and has a stable row count. endToEndInner :: Test endToEndInner = TestCase $ @@ -123,30 +108,22 @@ 600000 (D.nRows joined) -{- | A probe frame whose join key is TEXT and whose row count crosses the -parallel row-hash threshold, so the inner join hashes the key in parallel. -Exercises the parallel text/factor-key hash path (the medium-factor lever) end -to end: a factor key with @k@ distinct values mapped over @n@ rows. --} +-- | Probe frame with a text join key and enough rows for parallel hashing. factorOrdersFrame :: Int -> Int -> D.DataFrame factorOrdersFrame n k = - D.fromNamedColumns + D.fromColumns [ ("fk", D.fromList [T.pack ('f' : show (i `mod` k)) | i <- [0 .. n - 1]]) , ("amount", D.fromList [fromIntegral i * 2.0 :: Double | i <- [0 .. n - 1]]) ] factorDimFrame :: Int -> D.DataFrame factorDimFrame k = - D.fromNamedColumns + D.fromColumns [ ("fk", D.fromList [T.pack ('f' : show i) | i <- [0 .. k - 1]]) , ("label", D.fromList [T.pack ('L' : show i) | i <- [0 .. k - 1]]) ] -{- | Inner join on a TEXT key over a frame past the parallel-hash threshold: the -every-key-matches case must keep all @n@ probe rows, confirming the parallel -text hashing buckets identically to a sequential pass (a miscomputed parallel -hash would drop matches and shrink the row count). --} +-- | Inner join on a text key keeps all n probe rows under parallel hashing. endToEndFactorInner :: Test endToEndFactorInner = TestCase $ @@ -159,13 +136,7 @@ n (D.nRows joined) -{- | Small-build (cache-resident dim), very large probe (> parProbeThreshold) on -a TEXT/factor key, the medium-factor lever. The public 'innerJoin' takes the -parallel small-build-probe path here; every probe key matches a dim key, so a -mis-partitioned parallel probe that dropped or duplicated matches would change -the kept row count. The bit-identity of the parallel and sequential kernels is -pinned separately by 'innerKernelParity' on the @small-build-big-probe@ shape. --} +-- | Small-build, very large probe inner join on a text key keeps every match. endToEndSmallBuildFactorInner :: Test endToEndSmallBuildFactorInner = TestCase $ @@ -179,11 +150,7 @@ n (D.nRows joined) -{- | Small-build, large-probe LEFT join on the factor key: every probe row is -kept (matched or sentinel). Exercises the parallel small-build 'parLeftKernel' -through the public 'leftJoin'. With @k@ distinct dim keys and probe keys in -@[0, 2k)@, roughly half the probe rows miss and carry a Nothing. --} +-- | Small-build, large-probe left join keeps every probe row, matched or not. endToEndSmallBuildFactorLeft :: Test endToEndSmallBuildFactorLeft = TestCase $ diff --git a/tests/Operations/Provenance.hs b/tests/Operations/Provenance.hs --- a/tests/Operations/Provenance.hs +++ b/tests/Operations/Provenance.hs @@ -15,7 +15,7 @@ -- Base frame with no derived columns. base :: D.DataFrame base = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList [1 .. 5 :: Int]) , ("y", DI.fromList [2 .. 6 :: Int]) ] @@ -24,8 +24,6 @@ withZ :: D.DataFrame withZ = D.derive "z" (F.col @Int "x" + F.col "y") base --- ── insertColumn ────────────────────────────────────────────────────────────── - -- Inserting a new column must not wipe provenance of existing derived columns. insertPreservesProvenance :: Test insertPreservesProvenance = @@ -53,8 +51,6 @@ "sibling column w expression should be preserved" (M.member "w" (DI.derivingExpressions df3)) --- ── derive ──────────────────────────────────────────────────────────────────── - -- derive adds the expression to derivingExpressions. deriveTracksExpression :: Test deriveTracksExpression = @@ -86,8 +82,6 @@ (M.size (DI.derivingExpressions df)) ) --- ── deriveWithExpr ──────────────────────────────────────────────────────────── - -- deriveWithExpr should also track the expression. deriveWithExprTracksExpression :: Test deriveWithExprTracksExpression = @@ -98,8 +92,6 @@ (M.member "z" (DI.derivingExpressions df)) ) --- ── showDerivedExpressions ──────────────────────────────────────────────────── - -- A frame with no derived columns returns identity provenance for each column. showDerivedEmpty :: Test showDerivedEmpty = @@ -119,8 +111,6 @@ ("z" `elem` map fst (D.showDerivedExpressions withZ)) ) --- ── Semigroup (<>) provenance propagation ───────────────────────────────────── - -- Vertical merge preserves expressions from both sides. semiGroupPreservesLeft :: Test semiGroupPreservesLeft = @@ -172,12 +162,10 @@ (M.member "z" (DI.derivingExpressions (D.empty <> withZ))) ) --- ── Horizontal merge (|||) provenance propagation ──────────────────────────── - horizontalMergePreservesLeft :: Test horizontalMergePreservesLeft = let dfW = D.derive "w" (F.col @Int "y") base - extra = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] + extra = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] merged = dfW ||| extra in TestCase ( assertBool @@ -187,12 +175,12 @@ horizontalMergePreservesRight :: Test horizontalMergePreservesRight = - let extra = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] + let extra = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] dfW = D.derive "w" (F.col @Int "y") - (D.fromNamedColumns [("y", DI.fromList [2 .. 6 :: Int])]) + (D.fromColumns [("y", DI.fromList [2 .. 6 :: Int])]) merged = extra ||| dfW in TestCase ( assertBool @@ -203,7 +191,7 @@ horizontalMergePreservesBoth :: Test horizontalMergePreservesBoth = let dfZ = withZ - dfW = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] + dfW = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] -- give dfW a derived column on a separate base dfWD = D.derive "w" (F.col @Int "q") dfW merged = dfZ ||| dfWD diff --git a/tests/Operations/ReadCsv.hs b/tests/Operations/ReadCsv.hs --- a/tests/Operations/ReadCsv.hs +++ b/tests/Operations/ReadCsv.hs @@ -91,9 +91,7 @@ assertFailure "expected UnboxedColumn for 'girls' via fallback InferFromSample 10" --- File: a,b,c header; row "1,2,3,EXTRA" --- Total delimiters: 3 + 4 = 7; 7 div 3 = 2, numRow=1 --- Row-1 strides 3,4,5 → "1","2","3" — EXTRA (stride 6) is never accessed +-- Row "1,2,3,EXTRA": the extra cell (stride 6) is never accessed. testExtraFields :: Test testExtraFields = TestLabel "malformed_extra_fields" $ TestCase $ do df <- readCsvNoInfer ("./tests/data/unstable_csv/" <> "extra_fields.csv") @@ -107,9 +105,7 @@ (D.fromList @T.Text ["3"]) col --- Inference + EitherRead: first rows of "score" are clean ints, later rows are --- "abc" / empty / "N/A". The sample is limited to the clean prefix so the --- assumption is Int; non-parsing later rows become Left carrying raw text. +-- Inference + EitherRead: later non-parsing rows become Left with raw text. eitherReadCsvInference :: Test eitherReadCsvInference = TestLabel "csv_eitherRead_inference" $ TestCase $ do df <- @@ -149,8 +145,7 @@ expectedId col Nothing -> assertFailure "id column missing" - -- 'name' has a row containing "N/A"; under EitherRead the text is just - -- a value so it lands in Right. Empty cells would become Left "". + -- Under EitherRead 'name' text is just a value, so it lands in Right. let expectedName :: Column expectedName = D.fromList @(Either T.Text T.Text) @@ -170,9 +165,7 @@ D.readCsvWithOpts D.defaultReadOptions{D.safeRead = D.MaybeRead} "./tests/data/unstable_csv/either_read_mixed.csv" - -- "score" is not cleanly parseable as Int because "abc" fails; however the - -- BS inference walks all rows and falls back to Text when needed. The - -- relevant invariant for MaybeRead is that the column is nullable. + -- The invariant for MaybeRead is that the column is nullable. case getColumn "score" df of Just (UnboxedColumn (Just _) _) -> pure () -- Int with bitmap Just (BoxedColumn (Just _) _) -> pure () -- Text-backed with bitmap @@ -184,8 +177,7 @@ <> " with no bitmap" Nothing -> assertFailure "score column missing" --- NoSafeRead leaves columns un-bitmapped when every row parses (id column is --- all ints) but falls back to a Text column for mixed rows like "score". +-- NoSafeRead: all-int columns stay un-bitmapped; mixed rows fall back to Text. noSafeReadCsv :: Test noSafeReadCsv = TestLabel "csv_noSafeRead" $ TestCase $ do df <- @@ -200,9 +192,7 @@ <> columnTypeString col Nothing -> assertFailure "id column missing" --- Lazy CSV reader + EitherRead: goes through a different pipeline --- (MutableColumn + null indices). Every row of 'id' parses cleanly, so the --- column is @Either Text Int@ with every value in 'Right'. +-- Lazy CSV reader + EitherRead: 'id' is @Either Text Int@ with all Right. lazyEitherReadCsv :: Test lazyEitherReadCsv = TestLabel "lazy_csv_eitherRead" $ TestCase $ do (df, _) <- @@ -221,9 +211,7 @@ col Nothing -> assertFailure "id column missing (lazy reader)" --- Per-column overrides (eager): 'id' is strict Int (NoSafeRead), 'score' is --- Either Text Int (EitherRead) recording the raw bytes for failures, 'name' --- falls back to the default (MaybeRead) and becomes a nullable Text column. +-- Per-column overrides (eager): 'id' strict Int, 'score' Either, 'name' Maybe. perColumnEagerOverrides :: Test perColumnEagerOverrides = TestLabel "csv_perColumnOverrides_eager" $ TestCase $ do df <- @@ -265,9 +253,7 @@ expectedScore col Nothing -> assertFailure "score column missing" - -- 'name': default MaybeRead kicks in → nullable column (bitmap attached). - -- Every cell is non-empty so the bitmap is all-valid, but the optional - -- wrap is still applied. + -- 'name': default MaybeRead gives a nullable column with a bitmap. case getColumn "name" df of Just (BoxedColumn (Just _) _) -> pure () Just (PackedText (Just _) _) -> pure () @@ -277,8 +263,7 @@ <> columnTypeString col Nothing -> assertFailure "name column missing" --- Per-column overrides (lazy): same configuration as the eager test, but --- exercises the lazy freezer's resolution of per-column modes. +-- Per-column overrides (lazy): same configuration as the eager test. perColumnLazyOverrides :: Test perColumnLazyOverrides = TestLabel "csv_perColumnOverrides_lazy" $ TestCase $ do (df, _) <- @@ -300,9 +285,7 @@ "lazy NoSafeRead override for 'id' → bare UnboxedColumn, got " <> columnTypeString col Nothing -> assertFailure "id column missing (lazy)" - -- EitherRead on 'score': every row is Right (first row was '10' → Int - -- builder), though rows that failed parsing are captured as Left via the - -- nulls list. We only assert the column is Boxed-of-Either. + -- EitherRead on 'score': we only assert the column is Boxed-of-Either. case getColumn "score" df of Just (BoxedColumn Nothing _) -> pure () Just col -> @@ -311,8 +294,7 @@ <> columnTypeString col Nothing -> assertFailure "score column missing (lazy)" --- Default=NoSafeRead, override single column 'score' → MaybeRead. --- 'id' and 'name' keep the NoSafeRead default; 'score' becomes nullable. +-- Default NoSafeRead with 'score' overridden to MaybeRead (nullable). overrideNoSafeReadDefaultWithMaybeRead :: Test overrideNoSafeReadDefaultWithMaybeRead = TestLabel "csv_override_noSafeRead_default_MaybeRead" $ TestCase $ do @@ -334,17 +316,14 @@ -- 'score': MaybeRead override → nullable (bitmap present). case getColumn "score" df of Just (UnboxedColumn (Just _) _) -> pure () -- Int with bitmap - Just (BoxedColumn (Just _) _) -> pure () -- fallback Text with bitmap + Just (BoxedColumn (Just _) _) -> pure () Just (PackedText (Just _) _) -> pure () -- packed Text with bitmap Just col -> assertFailure $ "'score' MaybeRead override should yield nullable column, got " <> columnTypeString col Nothing -> assertFailure "score column missing" - -- 'name': NoSafeRead default → Text; may have bitmap when data has - -- null cells (row 8 has "N/A" which is in missingIndicators). - -- ensureOptional is NOT forced, but a bitmap is still created when - -- the builder detects actual nulls. + -- 'name': NoSafeRead default; a bitmap appears only if nulls exist. case getColumn "name" df of Just (BoxedColumn _ _) -> pure () Just (PackedText _ _) -> pure () @@ -519,8 +498,7 @@ "lazy 'id' NoSafeRead → bare UnboxedColumn, got " <> columnTypeString col Nothing -> assertFailure "id missing (lazy)" - -- 'score': EitherRead override → BoxedColumn of Either values. - -- Lazy reader infers Int from first row; rows that fail become Left. + -- 'score': EitherRead override gives BoxedColumn of Either values. case getColumn "score" df of Just (BoxedColumn Nothing _) -> pure () Just col -> diff --git a/tests/Operations/Record.hs b/tests/Operations/Record.hs --- a/tests/Operations/Record.hs +++ b/tests/Operations/Record.hs @@ -49,7 +49,7 @@ "./tests/data/alltypes_plain.parquet" ) --- Nullable fields (Maybe Text -> RNullableBoxed; Maybe Int -> RNullableUnboxed). +-- Nullable (Maybe Text -> RNullableBoxed, Maybe Int -> RNullableUnboxed). data User = User { userId :: Int64 , userName :: Maybe T.Text @@ -112,8 +112,8 @@ instance DT.HasSchema Foo where type Schema Foo = FooSchema - toColumns = DT.genericToColumns - fromColumns = DT.genericFromColumns + recordsToColumns = DT.genericToColumns + recordsFromFrame = DT.genericFromColumns orderSample :: [Order] orderSample = @@ -182,7 +182,7 @@ typeMismatchError :: Test typeMismatchError = TestCase $ do let badDf = - D.fromNamedColumns + D.fromColumns [ ("order_id", DI.fromList ([1, 2, 3] :: [Int])) , ("region", DI.fromList (["us", "eu", "ap"] :: [T.Text])) , ("amount", DI.fromList ([10.0, 20.5, 30.0] :: [Double])) @@ -197,7 +197,7 @@ missingColumnError :: Test missingColumnError = TestCase $ do let badDf = - D.fromNamedColumns + D.fromColumns [ ("region", DI.fromList (["us"] :: [T.Text])) , ("amount", DI.fromList ([10.0] :: [Double])) ] diff --git a/tests/Operations/RowWise.hs b/tests/Operations/RowWise.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/RowWise.hs @@ -0,0 +1,148 @@ +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE TypeApplications #-} + +-- | An expression means what it computes for one row at a time. +module Operations.RowWise (tests) where + +import Control.Exception (throw) +import qualified Data.Text as T +import qualified Data.Vector as V + +import qualified DataFrame as D +import DataFrame.Expression.Operators ((./=.), (.<=.), (.>.)) +import qualified DataFrame.Functions as F +import DataFrame.Internal.Column (Columnable, TypedColumn (..), toVector) +import DataFrame.Internal.Expression (Expr (..)) +import DataFrame.Internal.Interpreter (interpret) + +import Test.HUnit + +-- | The second size is above the 200k threshold where loops run in parallel. +sizes :: [Int] +sizes = [60, 200003] + +xs, ys :: Int -> [Int] +xs n = take n (cycle [7, -3, 0, 12, 5, -8, 1]) +ys n = take n (cycle [2, 0, -1, 0, 3]) + +ds :: Int -> [Double] +ds n = take n (cycle [0.5, -1.5, 2.25, 100, -0.0, 3]) + +names :: Int -> [T.Text] +names n = take n (cycle ["Eco", "Business", "", "Eco Plus"]) + +maybes :: Int -> [Maybe Int] +maybes n = take n (cycle [Just 1, Nothing, Just 3, Nothing, Just 5, Just 6]) + +frame :: Int -> D.DataFrame +frame n = + D.fromColumns + [ "x" D.=: xs n + , "y" D.=: ys n + , "d" D.=: ds n + , "name" D.=: names n + , "m" D.=: maybes n + , "flag" D.=: map (fmap (> 2)) (maybes n) + ] + +eval :: forall a. (Columnable a) => Int -> Expr a -> Either String [a] +eval n e = case interpret @a (frame n) e of + Left err -> Left (show err) + Right (TColumn col) -> Right (V.toList (either throw id (toVector @a @V.Vector col))) + +x, y :: Expr Int +x = F.col "x" +y = F.col "y" + +d :: Expr Double +d = F.col "d" + +-- | The expression's rows equal the reference rows, at every size. +rows :: (Columnable a) => String -> Expr a -> (Int -> [a]) -> [Test] +rows label e expected = + [ TestCase + (assertEqual (label ++ " n=" ++ show n) (Right (expected n)) (eval n e)) + | n <- sizes + ] + +tests :: [Test] +tests = + concat + [ rows + "a guard protects its branch" + (If (y ./=. F.lit 0) (F.div x y) (F.lit 0)) + (\n -> [if b /= 0 then a `div` b else 0 | (a, b) <- zip (xs n) (ys n)]) + , rows + "a guarded branch may repeat a failing operation" + (If (y ./=. F.lit 0) (F.div x y + F.div x y) (F.lit 0)) + (\n -> [if b /= 0 then 2 * (a `div` b) else 0 | (a, b) <- zip (xs n) (ys n)]) + , rows + "a null in the branch not chosen is ignored" + (If (x .>. F.lit 0) (F.col @(Maybe Int) "m") (F.lit (Just 0))) + (\n -> [if a > 0 then m else Just 0 | (a, m) <- zip (xs n) (maybes n)]) + , rows + "nested conditions" + ( If + (x .>. F.lit 0) + (If (y .>. F.lit 0) (x + y) (x - y)) + (If (y .>. F.lit 1) (x * y) x) + ) + ( \n -> + [ if a > 0 then (if b > 0 then a + b else a - b) else (if b > 1 then a * b else a) + | (a, b) <- zip (xs n) (ys n) + ] + ) + , rows + "an aggregate inside a branch sees every row" + (If (d .<=. F.lit 1) (d - F.mean d) (F.lit 0)) + ( \n -> + let mean = sum (ds n) / fromIntegral n + in [if v <= 1 then v - mean else 0 | v <- ds n] + ) + , rows + "text branches" + (If (x .>. F.lit 0) (F.lift T.toUpper (F.col @T.Text "name")) (F.lit "none")) + (\n -> [if a > 0 then T.toUpper t else "none" | (a, t) <- zip (xs n) (names n)]) + , rows + "nullable branches" + ( If + (x .>. F.lit 0) + (F.lift (fmap (+ 1)) (F.col @(Maybe Int) "m")) + (F.col @(Maybe Int) "m") + ) + (\n -> [if a > 0 then fmap (+ 1) m else m | (a, m) <- zip (xs n) (maybes n)]) + , rows + "a repeated subexpression" + ((x + y) * (x + y) - (x + y)) + (\n -> [(a + b) * (a + b) - (a + b) | (a, b) <- zip (xs n) (ys n)]) + , rows + "a subexpression repeated across branches" + (If (y .>. F.lit 0) ((x + y) * F.lit 2) ((x + y) * F.lit 3)) + (\n -> [if b > 0 then (a + b) * 2 else (a + b) * 3 | (a, b) <- zip (xs n) (ys n)]) + , rows + "different functions under one name are not merged" + (F.lift (+ 1) x + F.lift (* 2) x) + (\n -> [(a + 1) + (a * 2) | a <- xs n]) + , + [ TestCase + ( assertBool + "a missing column is reported even in a branch never chosen" + ( either + (const True) + (const False) + (eval 60 (If (F.lit True) x (F.col @Int "absent"))) + ) + ) + , TestCase + ( assertBool + "a null condition is an error" + ( either + (const True) + (const False) + (eval 60 (If (F.col @Bool "flag") x y)) + ) + ) + ] + ] diff --git a/tests/Operations/SetOps.hs b/tests/Operations/SetOps.hs --- a/tests/Operations/SetOps.hs +++ b/tests/Operations/SetOps.hs @@ -9,29 +9,27 @@ import Test.HUnit -{- | Sort by the integer key column so set results (which come out in -hash-bucket order) can be compared deterministically. --} +-- | Sort by the integer key so hash-ordered set results compare stably. sortByA :: D.DataFrame -> D.DataFrame sortByA = D.sortBy [D.Asc (F.col @Int "A")] dfA :: D.DataFrame dfA = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList [1 :: Int, 2, 3, 3]) , ("B", DI.fromList ['a', 'b', 'c', 'c']) ] dfB :: D.DataFrame dfB = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList [3 :: Int, 4]) , ("B", DI.fromList ['c', 'd']) ] expect :: [Int] -> [Char] -> D.DataFrame expect as bs = - D.fromNamedColumns + D.fromColumns [ ("A", DI.fromList as) , ("B", DI.fromList bs) ] diff --git a/tests/Operations/Shuffle.hs b/tests/Operations/Shuffle.hs --- a/tests/Operations/Shuffle.hs +++ b/tests/Operations/Shuffle.hs @@ -15,7 +15,7 @@ testDataFrame :: D.DataFrame testDataFrame = - D.fromNamedColumns + D.fromColumns [ ("numbers", D.fromList @Int [1 .. 26]) ] @@ -44,8 +44,7 @@ (D.columnNames testDataFrame) ) --- Test that un-shuffling restores the original dataframe --- which is known to be sorted in this case +-- Un-shuffling restores the original (sorted) dataframe. shufflePreservesData :: Test shufflePreservesData = let gen = mkStdGen 1234 @@ -54,8 +53,7 @@ in TestCase (assertEqual "sort recovers initial numbers" testDataFrame sortedShuffled) --- Test that shuffling isn't doing anything sneaky with summoning --- random numbers somehow +-- Shuffling must not summon random numbers somehow. shuffleSameSeedIsSameShuffle :: Test shuffleSameSeedIsSameShuffle = let gen = mkStdGen 1234 @@ -103,24 +101,13 @@ (shuffledIndices (mkStdGen 7) 1) ) -{- | Chi-squared statistic of observed counts against a flat expectation: -sum over cells of (observed - expected)^2 / expected. --} +-- | Chi-squared statistic of observed counts against a flat expectation. chiSquared :: [Int] -> Double chiSquared counts = let expected = fromIntegral (sum counts) / fromIntegral (length counts) in sum [(fromIntegral o - expected) ^ (2 :: Int) / expected | o <- counts] -{- | Every permutation of n items is equally likely under a uniform shuffle, -so the counts over all n! outcomes are chi-squared with n! - 1 degrees of -freedom. Testing the whole permutation, rather than one position at a time, -also catches a shuffle whose positions are individually uniform but -correlated. Seeds are fixed, so the sample -- and the verdict -- is -deterministic. - -n = 5 gives 120 outcomes; 12000 draws puts 100 in each on average. The bound -is the 0.999 quantile of chi-squared with 119 degrees of freedom. --} +-- | Every permutation of n items is equally likely under a uniform shuffle. shufflePermutationsAreUniform :: Test shufflePermutationsAreUniform = let n = 5 @@ -137,15 +124,7 @@ (stat < 172.4) ) -{- | The frequency test from Knuth 3.3.2: each item lands in each position with -probability 1/n, so the n x n position-by-item table is chi-squared with -(n - 1)^2 degrees of freedom. A larger n than the permutation test can afford, -to catch bias that only shows at scale, such as a shuffle that leaves a -suffix untouched or never leaves an item in place. - -n = 10 and 5000 draws put 500 in each cell. The bound is the 0.999 quantile of -chi-squared with 81 degrees of freedom. --} +-- | Knuth's frequency test: each item lands in each position with chance 1/n. shufflePositionsAreUniform :: Test shufflePositionsAreUniform = let n = 10 diff --git a/tests/Operations/Sort.hs b/tests/Operations/Sort.hs --- a/tests/Operations/Sort.hs +++ b/tests/Operations/Sort.hs @@ -23,11 +23,11 @@ ] testData :: D.DataFrame -testData = D.fromNamedColumns values +testData = D.fromColumns values moreTestData :: D.DataFrame moreTestData = - D.fromNamedColumns + D.fromColumns [ ("test1", DI.fromList $ replicate 10 (0 :: Int) ++ replicate 10 1) , ("test2", DI.fromList $ [1 :: Int .. 10] ++ [1 .. 10]) ] @@ -37,7 +37,7 @@ TestCase ( assertEqual "Sorting rows by ascending works as intended" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList [(1 :: Int) .. 26]) , ("test2", DI.fromList ['a' .. 'z']) ] @@ -50,7 +50,7 @@ TestCase ( assertEqual "Sorting rows by descending works as intended" - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList $ reverse [(1 :: Int) .. 26]) , ("test2", DI.fromList $ reverse ['a' .. 'z']) ] @@ -72,7 +72,7 @@ TestCase ( assertEqual "Sorting moreTestData by Desc of test2 reverses the order of the second column." - ( D.fromNamedColumns + ( D.fromColumns [ ("test1", DI.fromList $ replicate 10 (0 :: Int) ++ replicate 10 1) , ("test2", DI.fromList $ [10 :: Int, 9 .. 1] ++ [10, 9 .. 1]) ] @@ -91,7 +91,7 @@ compoundTestData :: D.DataFrame compoundTestData = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList ([3, 1, 4, 1, 5] :: [Int])) , ("b", DI.fromList ([10, 40, 20, 30, 50] :: [Int])) ] @@ -101,7 +101,7 @@ TestCase ( assertEqual "Sorting by Asc (a + b) orders rows by the sum without leaking a synthetic column" - ( D.fromNamedColumns + ( D.fromColumns [ ("a", DI.fromList ([3, 4, 1, 1, 5] :: [Int])) , ("b", DI.fromList ([10, 20, 30, 40, 50] :: [Int])) ] @@ -114,7 +114,7 @@ TestCase ( assertEqual "Sorting by Desc (b - a) orders rows by descending difference" - ( D.fromNamedColumns + ( D.fromColumns [ ("a", DI.fromList ([5, 1, 1, 4, 3] :: [Int])) , ("b", DI.fromList ([50, 40, 30, 20, 10] :: [Int])) ] @@ -127,7 +127,7 @@ TestCase ( assertEqual "Mixing a compound Asc key with a bare Desc tie-breaker works" - ( D.fromNamedColumns + ( D.fromColumns [ ("a", DI.fromList ([1, 1, 3, 4, 5] :: [Int])) , ("b", DI.fromList ([40, 30, 10, 20, 50] :: [Int])) ] diff --git a/tests/Operations/SortMask.hs b/tests/Operations/SortMask.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/SortMask.hs @@ -0,0 +1,102 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeApplications #-} + +-- | The radix sort and parallel mask paths agree with simple references. +module Operations.SortMask where + +import Data.Bits (shiftR, xor) +import Data.Function ((&)) +import Data.Int (Int32) +import qualified Data.List as L +import Data.Ord (Down (..), comparing) +import qualified Data.Vector.Unboxed as VU + +import qualified DataFrame as D +import qualified DataFrame.Functions as F +import DataFrame.Internal.Column (maskIndices) +import qualified DataFrame.Internal.Column as DI +import qualified DataFrame.Internal.DataFrame as DD +import Test.HUnit + +-- | Deterministic pseudo-random values in @[0, range)@. +draws :: Int -> Int -> Int -> [Int] +draws seed range n = [mix (seed * 1000003 + i) `mod` range | i <- [1 .. n]] + where + mix z0 = + let z1 = (z0 `xor` (z0 `shiftR` 30)) * (-4658895280553007687) + z2 = (z1 `xor` (z1 `shiftR` 27)) * (-7723592293110705685) + in abs (z2 `xor` (z2 `shiftR` 31)) + +-- | Row order of a stable sort, the order 'D.sortBy' must reproduce. +stableOrder :: (Ord k) => (a -> k) -> [a] -> [Int] +stableOrder key xs = map fst (L.sortBy (comparing (key . snd)) (zip [0 ..] xs)) + +sortedV :: D.DataFrame -> [Int] +sortedV df = maybe [] (DI.toList @Int) (DD.getColumn "v" df) + +-- Many ties and negative keys, so stability and sign handling both show. +intSort :: Bool -> Test +intSort desc = TestCase $ do + let n = 300000 + ks = map (subtract 500) (draws 7 1000 n) + df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. n - 1 :: Int])] + ord = if desc then D.Desc (F.col @Int "k") else D.Asc (F.col @Int "k") + expected = if desc then stableOrder Down ks else stableOrder id ks + assertEqual + ("Int sort desc=" ++ show desc) + expected + (sortedV (df & D.sortBy [ord])) + +narrowSort :: Test +narrowSort = TestCase $ do + let ks = map (fromIntegral . subtract 50) (draws 8 100 5000) :: [Int32] + df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. 4999 :: Int])] + assertEqual + "Int32 sort" + (stableOrder id ks) + (sortedV (df & D.sortBy [D.Asc (F.col @Int32 "k")])) + +-- -0.0 and 0.0 compare equal, so they keep row order. +doubleSort :: Test +doubleSort = TestCase $ do + let ks = + [0.0, -0.0, 1.5, -2.25, 1 / 0, -(1 / 0), 0.0, 1.5, -1.0e-300, 1.0e-300] :: + [Double] + df = + D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. length ks - 1])] + assertEqual + "Double sort" + (stableOrder id ks) + (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")])) + assertEqual + "Double sort desc" + (stableOrder Down ks) + (sortedV (df & D.sortBy [D.Desc (F.col @Double "k")])) + +-- A NaN column keeps the comparison sort; this only checks it still sorts. +nanSort :: Test +nanSort = TestCase $ do + let ks = [2, 0 / 0, 1, 3] :: [Double] + df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. 3 :: Int])] + assertEqual + "NaN sort length" + 4 + (length (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")]))) + +maskCases :: [Test] +maskCases = + [ TestCase $ + let bs = map ((< t) . (`mod` 10)) (draws n 10 n) + in assertEqual + ("maskIndices n=" ++ show n ++ " t=" ++ show t) + (VU.fromList [i | (i, True) <- zip [0 ..] bs]) + (maskIndices (VU.fromList bs)) + | n <- [0, 1, 1000, 300001] + , t <- [0, 3, 10] + ] + +tests :: [Test] +tests = + map + (TestLabel "sortMask") + ([intSort False, intSort True, narrowSort, doubleSort, nanSort] ++ maskCases) diff --git a/tests/Operations/Statistics.hs b/tests/Operations/Statistics.hs --- a/tests/Operations/Statistics.hs +++ b/tests/Operations/Statistics.hs @@ -191,7 +191,7 @@ 3 -- The three columns should be Statistics, A, and B ( D.nColumns ( D.summarize - ( D.fromNamedColumns + ( D.fromColumns [ ("A", D.fromList [1 :: Int, 2]) , ("B", D.fromList [Just (1 :: Int), Nothing]) ] @@ -200,13 +200,10 @@ ) ) -{- | A sliced column's bitmap keeps whole bytes, so its trailing bits still -describe rows past the end of the slice. Counting non-null rows has to stop -at the column's length rather than fold the whole byte vector. --} +-- | A sliced nullable column must count non-null rows only up to its length. sixteenNullableRows :: D.DataFrame sixteenNullableRows = - D.fromNamedColumns + D.fromColumns [ ("x", D.fromList (map (Just . fromIntegral) [1 .. 16 :: Int] :: [Maybe Double])) ] @@ -234,8 +231,7 @@ ) ) --- allMissing folds the same bitmap, so it needs the same length cutoff: the --- padding bits of a sliced all-null column would otherwise read as present. +-- allMissing folds the same bitmap, so it needs the same length cutoff. allMissingAfterSlice :: Test allMissingAfterSlice = TestCase @@ -247,14 +243,14 @@ allNullRows :: D.DataFrame allNullRows = - D.fromNamedColumns + D.fromColumns [("x", DI.fromList (replicate 16 (Nothing :: Maybe Double)))] -- correlation correlationDf :: D.DataFrame correlationDf = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList [1 :: Int, 2, 3, 4, 5]) , ("y_pos", DI.fromList [1 :: Int, 2, 3, 4, 5]) , ("y_neg", DI.fromList [5 :: Int, 4, 3, 2, 1]) diff --git a/tests/Operations/Subset.hs b/tests/Operations/Subset.hs --- a/tests/Operations/Subset.hs +++ b/tests/Operations/Subset.hs @@ -60,8 +60,7 @@ fst (dataframeDimensions (D.range (a, b) df)) == expected where rows = fst (dataframeDimensions df) - -- Rows in [a, b) that actually exist. Clamps before subtracting so the - -- oracle itself cannot overflow on extreme endpoints. + -- Rows in [a, b) that exist; clamps before subtracting to avoid overflow. lo = min (max a 0) rows hi = min (max b lo) rows expected = hi - lo @@ -96,7 +95,7 @@ prop_stratifiedSplit_deterministic :: DataFrame -> Bool prop_stratifiedSplit_deterministic _ = let df = - D.fromNamedColumns + D.fromColumns [ ("label", Col.fromList (replicate 50 ("A" :: T.Text) ++ replicate 50 "B")) , ("val", Col.fromList ([1 .. 100] :: [Int])) ] @@ -105,7 +104,7 @@ strataDf :: DataFrame strataDf = - D.fromNamedColumns + D.fromColumns [ ("label", Col.fromList (replicate 5 ("A" :: T.Text) ++ replicate 5 "B")) , ("val", Col.fromList ([1 .. 10] :: [Int])) ] @@ -132,7 +131,7 @@ unit_stratifiedSplit_singleRowStratum = TestCase $ let tinyDf = - D.fromNamedColumns + D.fromColumns [ ("label", Col.fromList (["A", "A", "A", "A", "A", "B"] :: [T.Text])) , ("val", Col.fromList ([1 .. 6] :: [Int])) ] @@ -142,7 +141,7 @@ (fst $ dataframeDimensions tinyDf) (fst (dataframeDimensions tr) + fst (dataframeDimensions va)) --- | Count occurrences of a label in a column, expressed as a fraction of total rows. +-- | Count of a label in a column, as a fraction of total rows. labelProportion :: T.Text -> T.Text -> DataFrame -> Double labelProportion col label df = let total = fst (dataframeDimensions df) @@ -158,7 +157,7 @@ let aCount = 100 bCount = 50 df = - D.fromNamedColumns + D.fromColumns [ ( "label" , Col.fromList (replicate aCount ("A" :: T.Text) ++ replicate bCount "B") @@ -191,7 +190,7 @@ (abs (vaProp - origProp) < tol) tenRows :: DataFrame -tenRows = fromNamedColumns [("x", Col.fromList ([0 .. 9] :: [Int]))] +tenRows = fromColumns [("x", Col.fromList ([0 .. 9] :: [Int]))] -- Endpoints that overflow Int if the length is computed before clamping. unit_rangeExtremeEndpoints :: Test diff --git a/tests/Operations/Take.hs b/tests/Operations/Take.hs --- a/tests/Operations/Take.hs +++ b/tests/Operations/Take.hs @@ -10,7 +10,7 @@ testData :: D.DataFrame testData = - D.fromNamedColumns + D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) ] diff --git a/tests/Operations/TextKernels.hs b/tests/Operations/TextKernels.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/TextKernels.hs @@ -0,0 +1,81 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeApplications #-} + +-- | contains and length on packed text agree with the Text functions per row. +module Operations.TextKernels (tests) where + +import qualified Data.ByteString as BS +import qualified Data.Text as T +import qualified Data.Text.Array as A +import qualified Data.Text.Encoding as TE +import Data.Text.Internal (Text (Text)) +import qualified Data.Vector.Unboxed as VU + +import qualified DataFrame as D +import qualified DataFrame.Functions as F +import qualified DataFrame.Internal.Column as DI +import DataFrame.Internal.Data.PackedText (mkPackedContiguous) +import Test.HUnit + +-- | Packed UTF-8 column of the given texts. +packed :: [T.Text] -> DI.Column +packed ts = + let bytes = BS.concat (map TE.encodeUtf8 ts) + offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts) + Text arr _ _ = TE.decodeUtf8 bytes + in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs)) + +rows :: [T.Text] +rows = + [ "http://www.google.com/search?q=1" + , "" + , "goo" + , "gle" + , "Google Docs" + , "ищу google здесь" + , "ééé.google.é" + , "a" + , "g" + , "gooogle" + ] + +frame :: D.DataFrame +frame = D.fromColumns [("t", packed rows)] + +containsCase :: T.Text -> Test +containsCase needle = + TestCase $ + assertEqual + ("contains " ++ show needle) + (map (T.isInfixOf needle) rows) + ( D.columnAsList + (F.col @Bool "m") + (D.derive "m" (F.contains needle (F.col "t")) frame) + ) + +lengthCase :: Test +lengthCase = + TestCase $ + assertEqual + "length" + (map T.length rows) + (D.columnAsList (F.col @Int "l") (D.derive "l" (F.length (F.col "t")) frame)) + +tests :: [Test] +tests = + map (TestLabel "text kernels") $ + lengthCase + : map + containsCase + [ "google" + , "Google" + , ".google." + , "" + , "g" + , "gle" + , "goo" + , "здесь" + , "é.g" + , "x" + , "gooogle!" + ] diff --git a/tests/Operations/TopBy.hs b/tests/Operations/TopBy.hs new file mode 100644 --- /dev/null +++ b/tests/Operations/TopBy.hs @@ -0,0 +1,51 @@ +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeApplications #-} + +-- | 'topBy' returns the window a full 'sortBy' would, for every key type. +module Operations.TopBy (tests) where + +import qualified Data.Text as T + +import qualified DataFrame as D +import qualified DataFrame.Functions as F +import Test.HUnit + +frame :: D.DataFrame +frame = + D.fromColumns + [ "n" D.=: [(i * 7919) `mod` 97 | i <- [0 .. 999 :: Int]] + , "d" + D.=: [fromIntegral ((i * 31) `mod` 13) / 4 :: Double | i <- [0 .. 999 :: Int]] + , "t" D.=: [T.pack (show ((i * 131) `mod` 53)) | i <- [0 .. 999 :: Int]] + , "id" D.=: [0 .. 999 :: Int] + ] + +orders :: [(String, [D.SortOrder])] +orders = + [ ("int asc", [D.Asc (F.col @Int "n")]) + , ("int desc", [D.Desc (F.col @Int "n")]) + , ("text asc", [D.Asc (F.col @T.Text "t")]) + , + ( "double desc, text asc" + , [D.Desc (F.col @Double "d"), D.Asc (F.col @T.Text "t")] + ) + , ("expr", [D.Asc (F.col @Int "n" * 3 - F.col @Int "id")]) + ] + +windows :: [(Int, Int)] +windows = + [(0, 1), (0, 10), (5, 10), (990, 10), (995, 20), (0, 1000), (0, 2000), (0, 0)] + +windowCases :: [Test] +windowCases = + [ TestCase $ + assertEqual + (label ++ " " ++ show (off, n)) + (D.range (off, off + n) (D.sortBy order frame)) + (D.topBy off n order frame) + | (label, order) <- orders + , (off, n) <- windows + ] + +tests :: [Test] +tests = map (TestLabel "topBy") windowCases diff --git a/tests/Operations/Typing.hs b/tests/Operations/Typing.hs --- a/tests/Operations/Typing.hs +++ b/tests/Operations/Typing.hs @@ -18,7 +18,7 @@ testData :: D.DataFrame testData = - D.fromNamedColumns + D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) ] @@ -71,8 +71,7 @@ datesInput :: [T.Text] datesInput = T.pack . show <$> datesExpected --- PARSING TESTS -------- 1. SIMPLE CASES +-- PARSING TESTS: 1. SIMPLE CASES parseBools :: Test parseBools = let afterParse :: [Bool] @@ -409,7 +408,7 @@ parseDatesAndEmptyStringsWithoutSafeRead :: Test parseDatesAndEmptyStringsWithoutSafeRead = - -- Pattern: 3 dates, empty, 3 dates, empty, 3 dates, empty, 3 dates, empty, 3 dates, empty + -- Pattern: 3 dates then an empty, repeated five times let groups = chunksOf3 datesExpected beforeParse = concatMap (\g -> map (T.pack . show) g ++ [""]) groups afterParse :: [Maybe Day] @@ -484,7 +483,7 @@ parseIntsAndDoublesAndNullishStringsWithoutSafeRead :: Test parseIntsAndDoublesAndNullishStringsWithoutSafeRead = - -- Nullish separators are literal text strings (not interpreted as null when safeRead=False) + -- Nullish separators are literal text when safeRead=False let nullishSeps = ["Nothing", "N/A", "NULL", "null", "NAN"] intGrp = T.pack . show <$> ([1 .. 10] :: [Int]) dblGrps = @@ -675,7 +674,7 @@ parseIntsAndDoublesAndNullishStringsWithSafeRead :: Test parseIntsAndDoublesAndNullishStringsWithSafeRead = - -- Note: this test uses different special doubles and nullish separators than the safeRead=False version + -- Uses different special doubles and nullish separators than safeRead=False let nullishSeps = ["Nothing", "N/A", "NULL", "null", "NAN"] intGrp = T.pack . show <$> ([1 .. 10] :: [Int]) dblGrps = @@ -736,7 +735,7 @@ parseIntsAndDoublesAndNullishAndEmptyStringsWithSafeRead :: Test parseIntsAndDoublesAndNullishAndEmptyStringsWithSafeRead = - -- Data: 4xN/A, "Nothing", "", then 10 ints (1-10), "", 10 ints (11-20), "", 10 ints (21-30), "", 3.14 + -- Data: 4xN/A, "Nothing", "", then ints 1-30 split by empties, 3.14 let beforeParse = ["N/A", "N/A", "N/A", "N/A", "Nothing", ""] ++ map (T.pack . show) ([1 .. 10] :: [Int]) @@ -770,7 +769,7 @@ parseTextsAndEmptyAndNullishStringsWithSafeRead = TestCase $ do raw <- T.lines <$> TIO.readFile (typingDataDir <> "texts_with_empties_and_nullish.txt") - -- safeRead=True: empty strings and nullish tokens (NaN, Nothing, N/A) -> Nothing + -- safeRead=True: empty strings and nullish tokens become Nothing let isNullish t = t `elem` ["NaN", "Nothing", "N/A", "nan", "null", "NULL", "NA", "na", "NAN"] afterParse = map (\t -> if t == "" || isNullish t then Nothing else Just t) raw expected = DI.fromVector $ V.fromList afterParse @@ -784,10 +783,6 @@ actual -- 3C. PARSING WITH SAFEREAD = EitherRead --- --- EitherRead wraps every column as @Either Text a@: successful parses become --- @Right v@; failures (including nullish/empty cells) become @Left @ --- preserving the original input verbatim. parseBoolsWithEitherRead :: Test parseBoolsWithEitherRead = @@ -840,7 +835,7 @@ parseDoublesWithEitherRead :: Test parseDoublesWithEitherRead = - -- Sample is cleanly Double; "oops" / "" come after and must surface as Left. + -- Sample is cleanly Double; later "oops" and "" must surface as Left. let beforeParse :: [T.Text] beforeParse = ["1.5", "2.0", "3.5", "4.25", "3.14", "oops", ""] afterParse :: [Either T.Text Double] @@ -908,12 +903,11 @@ actual ) --- parseDefaults honours per-column overrides: 'ages' stays strict Int, --- 'notes' is wrapped as Either Text Text. +-- parseDefaults honours per-column overrides of the default read mode. parseDefaultsPerColumnOverrides :: Test parseDefaultsPerColumnOverrides = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("ages", DI.fromList @T.Text ["1", "2", "3"]) , ("notes", DI.fromList @T.Text ["hello", "", "world"]) ] @@ -949,7 +943,7 @@ parseDefaultsNoSafeReadWithMaybeOverride :: Test parseDefaultsNoSafeReadWithMaybeOverride = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList @T.Text ["1", "2", "3"]) , ("y", DI.fromList @T.Text ["a", "", "c"]) ] @@ -979,7 +973,7 @@ parseDefaultsEitherReadWithNoSafeOverride :: Test parseDefaultsEitherReadWithNoSafeOverride = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("nums", DI.fromList @T.Text ["10", "20", "30"]) , ("tags", DI.fromList @T.Text ["foo", "", "baz"]) ] @@ -1010,7 +1004,7 @@ parseDefaultsEmptyOverrides :: Test parseDefaultsEmptyOverrides = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [("v", DI.fromList @T.Text ["1", "2", "3"])] opts1 = D.defaultParseOptions{D.sampleSize = 3} opts2 = opts1{D.parseSafeOverrides = []} @@ -1023,7 +1017,7 @@ parseDefaultsNonExistentOverride :: Test parseDefaultsNonExistentOverride = TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [("a", DI.fromList @T.Text ["1", "2", "3"])] opts = D.defaultParseOptions @@ -1042,8 +1036,7 @@ parseAllNullWithEitherRead :: Test parseAllNullWithEitherRead = - -- NoAssumption path: sample is all empty/nullish so inference can't - -- pick a numeric type. EitherRead still wraps as Either Text Text. + -- NoAssumption path: EitherRead still wraps as Either Text Text. let beforeParse :: [T.Text] beforeParse = ["", "", "", "N/A", ""] afterParse :: [Either T.Text T.Text] @@ -1212,7 +1205,7 @@ parseIntsAndDoublesAndEmptyStringsAsDoublesWithOneExampleWithSafeReadOff :: Test parseIntsAndDoublesAndEmptyStringsAsDoublesWithOneExampleWithSafeReadOff = - -- Pattern: 10 empties, 50 ints (as int text), 50 doubles (as double text), 5 specials + -- Pattern: 10 empties, 50 ints, 50 doubles, 5 specials let beforeParse = replicate 10 "" ++ intsInput @@ -1267,7 +1260,7 @@ parseIntsAndDoublesAndEmptyStringsAndNullishAsStringssWithOneExampleWithSafeReadOff :: Test parseIntsAndDoublesAndEmptyStringsAndNullishAsStringssWithOneExampleWithSafeReadOff = - -- Pattern: 10 empties, 5x(NaN,N/A), 20 ints (1-20), 30 doubles (1.0-30.0), 5 specials + -- Pattern: 10 empties, 5x(NaN,N/A), 20 ints, 30 doubles, 5 specials let nullishPairs = concat (replicate 5 ["NaN", "N/A"]) beforeParse = replicate 10 "" diff --git a/tests/Operations/VectorKernel.hs b/tests/Operations/VectorKernel.hs --- a/tests/Operations/VectorKernel.hs +++ b/tests/Operations/VectorKernel.hs @@ -2,31 +2,10 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | The vectorized scatter-accumulate kernel parity gate. Every case asserts -that 'D.aggregate' (which routes recognised reductions through the fast scatter -kernel) produces a result byte-identical to the same aggregation forced through -the expression interpreter ('interpretOnly'). Covered: sum/min/max/count over -Int and Double, mean, stddev/variance, top2Sum, median, the @max - min@ -compound (Q7), and a mixed multi-aggregation over single and multi-key -groupings, on Int/Double/nullable/Text-key grids and sizes straddling the -parallel threshold. - -Floating-point sums computed by the scatter follow the same left-to-right -group-order fold as the interpreter, so the two paths agree bit-for-bit here. -The PARALLEL path ('DataFrame.Internal.Aggregation.Kernel.Scatter') splits the work by -disjoint group-id range, and because each group's rows stay in their original -@valueIndices@ order within one worker's range, the per-group fold order is -unchanged from the sequential scatter — so the parallel path is also -byte-identical to the interpreter, asserted here at sizes above the 200k -parallel threshold (the test suite runs @-with-rtsopts=-N@, so those cases -exercise the multi-capability scatter). --} module Operations.VectorKernel where import Control.Exception (throw) -import qualified Data.List as L import qualified Data.Text as T -import qualified Data.Vector as V import qualified DataFrame as D import DataFrame.Errors (DataFrameException (..)) @@ -34,9 +13,8 @@ import qualified DataFrame.Internal.Column as DI import DataFrame.Internal.DataFrame (GroupedDataFrame, insertColumn) import DataFrame.Internal.Expression ( - AggStrategy (..), Expr (..), - UExpr (..), + SomeExpr (..), ) import DataFrame.Internal.Interpreter ( AggregationResult (..), @@ -48,11 +26,11 @@ import Test.HUnit -- | Aggregate forcing every expression through the interpreter (no kernel). -interpretOnly :: [(T.Text, UExpr)] -> GroupedDataFrame -> D.DataFrame +interpretOnly :: [(T.Text, SomeExpr)] -> GroupedDataFrame -> D.DataFrame interpretOnly aggs gdf = let base = DA.aggregate [] gdf - f :: (T.Text, UExpr) -> D.DataFrame -> D.DataFrame - f (name, UExpr (expr :: Expr a)) d = + f :: (T.Text, SomeExpr) -> D.DataFrame -> D.DataFrame + f (name, SomeExpr (expr :: Expr a)) d = let value :: DI.Column value = case interpretAggregation @a gdf expr of Left e -> throw e @@ -61,20 +39,12 @@ in insertColumn name value d in foldl (flip f) base aggs --- | Per-group sum of the two largest values (the benchmark's Q8 reduction). -top2Sum :: Expr Double -> Expr Double -top2Sum = Agg (CollectAgg "top2Sum" f) - where - f :: V.Vector Double -> Double - f v = sum (take 2 (L.sortBy (flip compare) (V.toList v))) - grid :: Int -> D.DataFrame grid n = - D.fromNamedColumns + D.fromColumns [ ("ki", DI.fromList [i `mod` 17 | i <- [0 .. n - 1]]) , ("kt", DI.fromList [T.pack ('g' : show (i `mod` 11)) | i <- [0 .. n - 1]]) - , -- A high-cardinality key so the parallel group-range split sees many - -- small groups (alongside the low-cardinality 'ki'/'kt' few-big-groups). + , -- A high-cardinality key so the group-range split sees many groups. ("kh", DI.fromList [(i * 2654435761) `mod` 50021 :: Int | i <- [0 .. n - 1]]) , ("vi", DI.fromList [(i * 31 + 7) `mod` 1000 - 500 :: Int | i <- [0 .. n - 1]]) , ("vj", DI.fromList [(i * 13) `mod` 97 :: Int | i <- [0 .. n - 1]]) @@ -91,7 +61,7 @@ vd :: Expr Double vd = F.col @Double "vd" -aggSets :: [[(T.Text, UExpr)]] +aggSets :: [[(T.Text, SomeExpr)]] aggSets = [ ["s" F..= F.sum vi] , ["s" F..= F.sum vd, "c" F..= F.count vi] @@ -99,7 +69,6 @@ , ["mnd" F..= F.minimum vd, "mxd" F..= F.maximum vd] , ["m" F..= F.mean vi, "md" F..= F.mean vd] , ["sd" F..= F.stddev vd, "var" F..= F.variance vd] - , ["t2" F..= top2Sum vd] , ["med" F..= F.median vd] , ["diff" F..= (F.maximum vi - F.minimum vj)] , @@ -115,7 +84,7 @@ keySets :: [[T.Text]] keySets = [["ki"], ["kt"], ["ki", "kt"], ["kh"]] -parityCase :: Int -> [T.Text] -> [(T.Text, UExpr)] -> Test +parityCase :: Int -> [T.Text] -> [(T.Text, SomeExpr)] -> Test parityCase n keys aggs = TestCase $ let df = grid n diff --git a/tests/Operations/Window.hs b/tests/Operations/Window.hs --- a/tests/Operations/Window.hs +++ b/tests/Operations/Window.hs @@ -13,10 +13,10 @@ import DataFrame.Expression.Operators import Test.HUnit --- Similar to example discussed in https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ +-- Similar to the example in sumsar.net's pandas-feels-clunky-from-r post. purchases :: D.DataFrame purchases = - D.fromNamedColumns + D.fromColumns [ ( "country" , DI.fromList @@ -57,7 +57,7 @@ ) & D.sortBy [D.Asc (F.col @T.Text "country"), D.Asc (F.col @Double "amount")] expected = - D.fromNamedColumns + D.fromColumns [ ( "country" , DI.fromList (["France", "France", "France", "UK", "UK", "US", "US"] :: [T.Text]) @@ -99,7 +99,7 @@ ) where simpleData = - D.fromNamedColumns + D.fromColumns [ ("group", DI.fromList (["A", "A", "B", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([10, 20, 30, 60, 90] :: [Double])) ] @@ -123,7 +123,7 @@ ) where simpleData = - D.fromNamedColumns + D.fromColumns [ ("group", DI.fromList (["X", "X", "Y", "Y"] :: [T.Text])) , ("value", DI.fromList ([10, 20, 100, 200] :: [Int])) ] @@ -146,7 +146,7 @@ ) where simpleData = - D.fromNamedColumns + D.fromColumns [ ("group", DI.fromList (["A", "A", "A", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([1, 2, 3, 4, 5] :: [Int])) ] @@ -169,7 +169,7 @@ ) where simpleData = - D.fromNamedColumns + D.fromColumns [ ("group", DI.fromList (["A", "A", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([10.0, 20.0, 100.0, 200.0] :: [Double])) ] @@ -184,7 +184,7 @@ Nothing -> error "deviation column not found" Just c -> DI.toList @Double c --- | Example from https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/ +-- | Example from the sumsar.net pandas-feels-clunky-from-r post. blogPostExampleGlobal :: Test blogPostExampleGlobal = TestCase @@ -202,10 +202,9 @@ & D.groupBy ["country"] & D.aggregate [F.sum (amount - discount) `as` "total"] & D.sortBy [D.Asc (F.col @T.Text "country")] - -- Global median = 60, threshold = 600 → removes 5000 and 800 - -- France: (30-1)+(40-2)+(35-1) = 101, UK: (50-2)+(60-3) = 105, US: (100-5)+(200-10) = 285 + -- Global median 60 and threshold 600 remove 5000 and 800. expected = - D.fromNamedColumns + D.fromColumns [ ("country", DI.fromList (["France", "UK", "US"] :: [T.Text])) , ("total", DI.fromList ([101, 105, 285] :: [Double])) ] @@ -228,7 +227,7 @@ & D.aggregate [F.sum (amount - discount) `as` "total"] & D.sortBy [D.Asc (F.col @T.Text "country")] expected = - D.fromNamedColumns + D.fromColumns [ ("country", DI.fromList (["France", "UK", "US"] :: [T.Text])) , ("total", DI.fromList ([101, 105, 285] :: [Double])) ] diff --git a/tests/Operations/WriteCsv.hs b/tests/Operations/WriteCsv.hs --- a/tests/Operations/WriteCsv.hs +++ b/tests/Operations/WriteCsv.hs @@ -15,7 +15,7 @@ toCsvBasic :: Test toCsvBasic = TestLabel "toCsv_basic" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("name", DI.fromList @T.Text ["Alice", "Bob", "Charlie"]) , ("age", DI.fromList @Int [30, 25, 35]) ] @@ -25,7 +25,7 @@ toCsvWithNulls :: Test toCsvWithNulls = TestLabel "toCsv_withNulls" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ( "name" , DI.fromList @(Maybe T.Text) [Nothing, Just "Alice", Just "Bob", Just "Charlie"] @@ -46,7 +46,7 @@ toSeparatedTab :: Test toSeparatedTab = TestLabel "toSeparated_tab" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList @Int [1, 2]) , ("y", DI.fromList @Int [3, 4]) ] @@ -57,7 +57,7 @@ toCsvDouble :: Test toCsvDouble = TestLabel "toCsv_double" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("value", DI.fromList @Double [1.5, 2.0, 2.5]) ] expected = "value\n1.5\n2.0\n2.5\n" @@ -67,7 +67,7 @@ toCsvSingleColumn :: Test toCsvSingleColumn = TestLabel "toCsv_single_column" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("id", DI.fromList @Int [10, 20, 30]) ] expected = "id\n10\n20\n30\n" @@ -77,7 +77,7 @@ toCsvRoundTrip :: Test toCsvRoundTrip = TestLabel "toCsv_roundTrip" $ TestCase $ do let df = - D.fromNamedColumns + D.fromColumns [ ("a", DI.fromList @Int [1, 2, 3]) , ("b", DI.fromList @T.Text ["hello", "world", "test"]) ] diff --git a/tests/PackedTextMigration.hs b/tests/PackedTextMigration.hs --- a/tests/PackedTextMigration.hs +++ b/tests/PackedTextMigration.hs @@ -1,13 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Regression tests for the PackedText migration: a string column stored as a -'DI.PackedText' (the form the CSV readers emit) must flow through the -decision-tree feature/target extraction and the full-outer-join key coalescing -without crashing, and must behave IDENTICALLY to the same data held as a boxed -@Text@ column. If a non-exhaustive @Column@ match regresses, the packed cases -either crash (test error) or diverge from the boxed baseline (assertion fails). --} +-- | Packed-text columns must behave identically to boxed Text columns. module PackedTextMigration (tests) where import Control.Monad (zipWithM_) @@ -29,9 +23,7 @@ import Test.HUnit -{- | Build a 'DI.PackedText' column directly from raw UTF-8 bytes + offsets, -exactly as the CSV readers do (mirrors @Internal.PackedText.packedFromTexts@). --} +-- | Build a packed-text column directly from UTF-8 bytes and offsets. packedFromTexts :: [T.Text] -> DI.Column packedFromTexts ts = let bytess = map (B.unpack . encodeUtf8) ts @@ -52,15 +44,11 @@ packed = packedFromTexts boxed = DI.fromList -{- | A decision tree using a TEXT FEATURE (exercises Cart.featuresOfColumn and the -categorical condition builders). The packed-feature tree must equal the -boxed-feature tree, proving the feature is used identically — not dropped or -crashed on. --} +-- | A decision tree on a packed text feature must equal the boxed-feature tree. treeOnTextFeature :: Test treeOnTextFeature = TestCase $ do let df mk = - D.fromNamedColumns + D.fromColumns [ ("color", mk ["red", "red", "blue", "blue", "green", "green"]) , ("y", DI.fromList ([0, 0, 1, 1, 2, 2] :: [Double])) ] @@ -71,13 +59,11 @@ (D.prettyPrint (predict boxedTree)) (D.prettyPrint (predict packedTree)) -{- | A classifier whose TARGET is a packed-text column (exercises -Cart.cartTargetLabels). Packed target must give the same tree as a boxed one. --} +-- | A classifier with a packed-text target gives the same tree as a boxed one. classifierOnTextTarget :: Test classifierOnTextTarget = TestCase $ do let df mk = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList ([1, 2, 3, 4, 5, 6] :: [Double])) , ("label", mk ["a", "a", "a", "b", "b", "b"]) ] @@ -88,17 +74,14 @@ (D.prettyPrint (predict boxedClf)) (D.prettyPrint (predict packedClf)) -{- | A full outer join on a packed-text KEY column (exercises the -coalesceKeyColumn path that previously errored on PackedText). Result must -match the boxed-key join and recover all four distinct keys. --} +-- | A full outer join on a packed-text key matches the boxed-key join. fullOuterJoinOnTextKey :: Test fullOuterJoinOnTextKey = TestCase $ do let l mk = - D.fromNamedColumns + D.fromColumns [("k", mk ["a", "b", "c"]), ("lv", DI.fromList ([1, 2, 3] :: [Double]))] r mk = - D.fromNamedColumns + D.fromColumns [("k", mk ["b", "c", "d"]), ("rv", DI.fromList ([20, 30, 40] :: [Double]))] packedJoin = fullOuterJoin ["k"] (l packed) (r packed) boxedJoin = fullOuterJoin ["k"] (l boxed) (r boxed) diff --git a/tests/Parquet.hs b/tests/Parquet.hs --- a/tests/Parquet.hs +++ b/tests/Parquet.hs @@ -146,7 +146,7 @@ TestCase ( assertEqual "predicateWithOpts" - (D.fromNamedColumns [("id", D.fromList [6 :: Int32, 7])]) + (D.fromColumns [("id", D.fromList [6 :: Int32, 7])]) ( unsafePerformIO ( D.readParquetWithOpts ( D.defaultParquetReadOptions @@ -169,7 +169,7 @@ TestCase ( assertEqual "predicateUsesNonSelectedColumnWithOpts" - (D.fromNamedColumns [("bool_col", D.fromList [True, False])]) + (D.fromColumns [("bool_col", D.fromList [True, False])]) ( unsafePerformIO ( D.readParquetWithOpts ( D.defaultParquetReadOptions @@ -353,10 +353,6 @@ where value = 0xA1B2C3D4 :: Word32 --- --------------------------------------------------------------------------- --- Group 1: Plain variant --- --------------------------------------------------------------------------- - allTypesTinyPagesPlain :: Test allTypesTinyPagesPlain = TestCase @@ -371,15 +367,7 @@ ) ) --- --------------------------------------------------------------------------- --- Group 2: Compression codecs (unsupported → error tests) --- --------------------------------------------------------------------------- - --- TODO: LZ4 and LZ4_RAW compression are not yet implemented. When support --- is added via a Haskell lz4 binding, hadoopLz4Compressed, --- hadoopLz4CompressedLarger, nonHadoopLz4Compressed, lz4RawCompressed, and --- lz4RawCompressedLarger should all change from assertExpectException to --- assertEqual checking their respective row/column dimensions. +-- TODO: Once LZ4/LZ4_RAW are implemented, the lz4 tests become assertEqual. hadoopLz4Compressed :: Test hadoopLz4Compressed = TestCase @@ -425,9 +413,7 @@ (D.readParquet "./tests/data/lz4_raw_compressed_larger.parquet") ) --- Was: assertExpectException "concatenatedGzipMembers" "12" ... --- The old parser failed with a ZLIB size error. The new decompressor --- handles concatenated gzip members correctly. +-- The decompressor now handles concatenated gzip members. concatenatedGzipMembers :: Test concatenatedGzipMembers = TestCase @@ -442,9 +428,7 @@ ) ) --- TODO: BROTLI compression is not yet implemented. When a Haskell brotli --- binding is added, change this to assertEqual checking the actual --- dimensions of large_string_map.brotli.parquet. +-- TODO: BROTLI is not implemented; change this to assertEqual once it is. largeBrotliMap :: Test largeBrotliMap = TestCase @@ -454,18 +438,7 @@ (D.readParquet "./tests/data/large_string_map.brotli.parquet") ) --- --------------------------------------------------------------------------- --- Group 3: Delta / RLE encodings (unsupported → error tests) --- --------------------------------------------------------------------------- - --- Was: assertExpectException "deltaBinaryPacked" "EDELTA_BINARY_PACKED" ... --- The new parser's error includes the encoding name "DELTA_BINARY_PACKED" --- without the old "E" prefix used in the previous error format. --- TODO: When DELTA_BINARY_PACKED (encoding id=5) is implemented, change --- this to assertEqual checking actual dimensions. The encoding stores --- integer data as bit-packed deltas and is common for monotonically --- increasing columns (row IDs, timestamps): --- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-encoding-delta_binary_packed--5 +-- TODO: Once DELTA_BINARY_PACKED is implemented, assert actual dimensions. deltaBinaryPacked :: Test deltaBinaryPacked = TestCase @@ -475,12 +448,7 @@ (D.readParquet "./tests/data/delta_binary_packed.parquet") ) --- Was: assertExpectException "deltaByteArray" "EDELTA_BYTE_ARRAY" ... --- Same reason as deltaBinaryPacked: new error format drops the "E" prefix. --- TODO: When DELTA_BYTE_ARRAY (encoding id=7) is implemented, change this --- to assertEqual checking actual dimensions. The encoding prefix-differences --- consecutive string values, reducing storage for sorted byte arrays: --- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-strings-delta_byte_array--7 +-- TODO: Once DELTA_BYTE_ARRAY is implemented, assert actual dimensions. deltaByteArray :: Test deltaByteArray = TestCase @@ -490,12 +458,7 @@ (D.readParquet "./tests/data/delta_byte_array.parquet") ) --- Was: assertExpectException "deltaEncodingOptionalColumn" "EDELTA_BINARY_PACKED" ... --- The first column that errors in this file uses DELTA_BYTE_ARRAY encoding, --- so we match the broader "unsupported encoding" substring instead. --- TODO: Once DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY are both implemented, --- change this to assertEqual checking the actual row count of --- delta_encoding_optional_column.parquet. +-- TODO: Once both DELTA encodings are implemented, assert the row count. deltaEncodingOptionalColumn :: Test deltaEncodingOptionalColumn = TestCase @@ -505,10 +468,7 @@ (D.readParquet "./tests/data/delta_encoding_optional_column.parquet") ) --- Was: assertExpectException "deltaEncodingRequiredColumn" "EDELTA_BINARY_PACKED" ... --- Same as deltaEncodingOptionalColumn: first failing column uses DELTA_BYTE_ARRAY. --- TODO: Same as deltaEncodingOptionalColumn — change to assertEqual once --- DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY encodings are both supported. +-- TODO: Once both DELTA encodings are implemented, change to assertEqual. deltaEncodingRequiredColumn :: Test deltaEncodingRequiredColumn = TestCase @@ -518,14 +478,7 @@ (D.readParquet "./tests/data/delta_encoding_required_column.parquet") ) --- Was: assertExpectException "deltaLengthByteArray" "ZSTD" ... --- The old parser failed during ZSTD decompression. The new parser --- detects the unsupported DELTA_LENGTH_BYTE_ARRAY encoding before decompression. --- TODO: When DELTA_LENGTH_BYTE_ARRAY (encoding id=6) is implemented, change --- this to assertEqual checking actual dimensions. The encoding stores a --- delta-encoded list of byte-array lengths followed by the raw concatenated --- values: --- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-length-byte-array-delta_length_byte_array--6 +-- TODO: Once DELTA_LENGTH_BYTE_ARRAY is implemented, assert dimensions. deltaLengthByteArray :: Test deltaLengthByteArray = TestCase @@ -535,13 +488,7 @@ (D.readParquet "./tests/data/delta_length_byte_array.parquet") ) --- Was: assertExpectException "rleBooleanEncoding" "Zlib" ... --- The old parser failed during Zlib decompression. The new parser --- detects the unsupported RLE boolean encoding before reaching decompression. --- TODO: When RLE/Bit-Packing Hybrid (encoding id=3, bit-width=1) is --- implemented for BOOLEAN columns, change this to assertEqual checking the --- actual decoded boolean values. The encoding is spec-valid for BOOLEAN: --- https://parquet.apache.org/docs/file-format/data-pages/encodings/#run-length-encoding--bit-packing-hybrid-rle--3 +-- TODO: Once RLE boolean encoding is implemented, assert the decoded values. rleBooleanEncoding :: Test rleBooleanEncoding = TestCase @@ -551,15 +498,7 @@ (D.readParquet "./tests/data/rle_boolean_encoding.parquet") ) --- Was: assertExpectException "dictPageOffsetZero" "Unknown kv" ... --- The old parser reported "Unknown kv" for a bad key-value field. The new --- Pinch-based page-header parser reports "Field 1 is absent" for the --- malformed page header in this file. --- TODO: Investigate whether dict-page-offset-zero.parquet can be read --- successfully with a more lenient page-header parser. If the missing --- mandatory field can be treated as a per-page soft error rather than --- aborting the whole read, this test would change to assertEqual --- checking actual dimensions. +-- TODO: Check whether a lenient page-header parser can read this file. dictPageOffsetZero :: Test dictPageOffsetZero = TestCase @@ -569,22 +508,7 @@ (D.readParquet "./tests/data/dict-page-offset-zero.parquet") ) --- --------------------------------------------------------------------------- --- Group 4: Data Page V2 (unsupported → error tests) --- --------------------------------------------------------------------------- - --- Was: assertExpectException "datapageV2Snappy" "InvalidOffset" ... --- The old parser failed with an offset validation error. The new parser --- first encounters the unsupported RLE encoding used by data-page-v2. --- TODO: Full Data Page V2 support requires two changes: --- 1. RLE/Bit-Packing Hybrid (id=3, bit-width=1) for BOOLEAN values --- (shared with rleBooleanEncoding above). --- 2. Parsing DataPageHeaderV2's in-line level streams: in v2, definition --- and repetition levels are stored uncompressed before the (optionally --- compressed) value bytes, with lengths given by --- definition_levels_byte_length and repetition_levels_byte_length. --- Once both are done, change to assertEqual checking actual dimensions: --- https://parquet.apache.org/docs/file-format/data-pages/ +-- TODO: Data Page V2 needs RLE booleans and in-line level stream parsing. datapageV2Snappy :: Test datapageV2Snappy = TestCase @@ -594,19 +518,7 @@ (D.readParquet "./tests/data/datapage_v2.snappy.parquet") ) --- Was: assertExpectException "datapageV2EmptyDatapage" "UnexpectedEOF" ... --- The old Snappy decompressor raised "UnexpectedEOF". The new Snappy --- library raises "EmptyInput" when given zero-length compressed data. --- The v2 page structure is parsed correctly: readLevelsV2V strips the --- in-line level streams before decompression, leaving an empty value --- payload (BS.empty) for a page with 0 values. The Snappy decompressor --- then raises "EmptyInput" because it is handed zero bytes. --- TODO: An empty data page (0 values) is valid and should contribute --- 0 rows without raising an error. The fix is a single guard in the --- DATA_PAGE_V2 branch of readPages (Page.hs): short-circuit --- decompressData when compValBytes is empty, returning BS.empty --- directly. Once fixed, change this to assertEqual checking the --- total expected row count of the file. +-- TODO: Guard empty V2 data pages in readPages so they yield 0 rows. datapageV2EmptyDatapage :: Test datapageV2EmptyDatapage = TestCase @@ -616,9 +528,7 @@ (D.readParquet "./tests/data/datapage_v2_empty_datapage.snappy.parquet") ) --- Was: assertExpectException "pageV2EmptyCompressed" "10" ... --- The old parser failed on empty compressed page-v2 blocks. The new parser --- treats empty compressed data as zero-value pages and reads all 10 rows. +-- Empty compressed page-v2 data is treated as zero-value pages. pageV2EmptyCompressed :: Test pageV2EmptyCompressed = TestCase @@ -633,10 +543,6 @@ ) ) --- --------------------------------------------------------------------------- --- Group 5: Checksum files (all read successfully) --- --------------------------------------------------------------------------- - datapageV1UncompressedChecksum :: Test datapageV1UncompressedChecksum = TestCase @@ -693,12 +599,7 @@ ) ) --- TODO: CRC checksum validation is not yet implemented; corrupt page --- checksums are silently ignored. When validation is added, consider a --- validateChecksums :: Bool field in ParquetReadOptions (default False) --- so callers can opt in. Once implemented, datapageV1CorruptChecksum and --- rleDictUncompressedCorruptChecksum should change to assertExpectException --- checking for a checksum mismatch error. +-- TODO: CRC checksum validation is not implemented; corrupt ones are ignored. datapageV1CorruptChecksum :: Test datapageV1CorruptChecksum = TestCase @@ -727,10 +628,6 @@ ) ) --- --------------------------------------------------------------------------- --- Group 6: NULL handling --- --------------------------------------------------------------------------- - nullsSnappy :: Test nullsSnappy = TestCase @@ -797,10 +694,6 @@ ) ) --- --------------------------------------------------------------------------- --- Group 7: Decimal types --- --------------------------------------------------------------------------- - int32Decimal :: Test int32Decimal = TestCase @@ -823,6 +716,19 @@ ) ) +nullableIntDecimal :: Test +nullableIntDecimal = testBothReadParquetPaths $ \readParquet -> + TestCase + ( assertEqual + "nullableIntDecimal" + ( D.fromColumns + [ ("int32_dec", D.fromList [Just (1 :: Double), Nothing, Just 2.5, Just 0.75]) + , ("int64_dec", D.fromList [Nothing, Just (1234.5 :: Double), Nothing, Just (-5)]) + ] + ) + (unsafePerformIO (readParquet "./tests/data/nullable_int_decimal.parquet")) + ) + byteArrayDecimal :: Test byteArrayDecimal = TestCase @@ -834,16 +740,7 @@ ) ) --- Was: assertExpectException "fixedLengthDecimal" "FIXED_LEN_BYTE_ARRAY" ... --- The old parser recognised FIXED_LEN_BYTE_ARRAY as a physical type but --- had no page decoder for it; reading data from such a column threw an --- error at the decoding stage. The new parser's fixedLenByteArrayDecoder --- reads the raw bytes and surfaces them as a text column. --- TODO: When the DECIMAL logical type is properly decoded for --- FIXED_LEN_BYTE_ARRAY columns, replace this dimension-only check with a --- value-level assertion verifying the actual decimal values (e.g. as --- Scientific or Double). The raw-byte Text column should become a typed --- numeric column. +-- TODO: Once DECIMAL is decoded for FIXED_LEN_BYTE_ARRAY, assert values. fixedLengthDecimal :: Test fixedLengthDecimal = TestCase @@ -855,11 +752,7 @@ ) ) --- Was: assertExpectException "fixedLengthDecimalLegacy" "FIXED_LEN_BYTE_ARRAY" ... --- Same as fixedLengthDecimal: the old parser had no page decoder for --- FIXED_LEN_BYTE_ARRAY; the new parser's fixedLenByteArrayDecoder handles it. --- TODO: Same as fixedLengthDecimal — add a value-level assertion once --- DECIMAL decoding over FIXED_LEN_BYTE_ARRAY is implemented. +-- TODO: Add a value-level assertion once DECIMAL decoding is implemented. fixedLengthDecimalLegacy :: Test fixedLengthDecimalLegacy = TestCase @@ -874,10 +767,6 @@ ) ) --- --------------------------------------------------------------------------- --- Group 8: Binary / fixed-length bytes --- --------------------------------------------------------------------------- - binaryFile :: Test binaryFile = TestCase @@ -903,9 +792,7 @@ ) ) --- Was: assertExpectException "fixedLengthByteArray" "FIXED_LEN_BYTE_ARRAY" ... --- Same as fixedLengthDecimal: the old parser had no page decoder for --- FIXED_LEN_BYTE_ARRAY; the new parser's fixedLenByteArrayDecoder handles it. +-- FIXED_LEN_BYTE_ARRAY is read by fixedLenByteArrayDecoder as raw text. fixedLengthByteArray :: Test fixedLengthByteArray = TestCase @@ -917,10 +804,6 @@ ) ) --- --------------------------------------------------------------------------- --- Group 9: INT96 timestamps --- --------------------------------------------------------------------------- - int96FromSpark :: Test int96FromSpark = TestCase @@ -932,13 +815,7 @@ ) ) --- --------------------------------------------------------------------------- --- Group 10: Metadata / index / bloom filters --- --------------------------------------------------------------------------- - --- Was: assertExpectException "columnChunkKeyValueMetadata" "Unknown page header field" ... --- The old parser rejected extra fields in page headers. Pinch ignores --- unknown fields gracefully. This file contains 0 data rows. +-- Unknown page header fields are ignored; this file has 0 data rows. columnChunkKeyValueMetadata :: Test columnChunkKeyValueMetadata = TestCase @@ -981,15 +858,7 @@ ) ) --- Was: assertEqual "sortColumns" (3, 2) ... --- The file contains two row groups, each storing 3 rows (6 rows total). --- DuckDB's parquet-metadata output shows row_group_num_rows=3, which is --- the count *per row group*, not the file total.row group*, not the file total.row group*, not the file total.row group*, not the file total. --- https://github.com/apache/parquet-testing/blob/master/data/README.md#:~:text=sort_columns.parquet --- The above link is to the repository the test parquet files comes from. --- The table describes sort_columns.parquet as having two row groups. --- The old parser only read the first row group (a bug). The new parser --- reads all row groups and returns (6, 2) correctly. +-- Two row groups of 3 rows each: all row groups are read, giving (6, 2). sortColumns :: Test sortColumns = TestCase @@ -1001,9 +870,7 @@ ) ) --- Was: assertExpectException "overflowI16PageCnt" "UNIMPLEMENTED" ... --- The old parser used Int16 for page counts and overflowed on this file. --- The new parser uses Int32 and reads all 40,000 rows correctly. +-- The parser uses Int32 page counts and reads all 40,000 rows. overflowI16PageCnt :: Test overflowI16PageCnt = TestCase @@ -1015,18 +882,7 @@ ) ) --- --------------------------------------------------------------------------- --- Group 11: Nested / complex types and byte-stream-split --- --------------------------------------------------------------------------- - --- Was: assertExpectException "byteStreamSplitZstd" "EBYTE_STREAM_SPLIT" ... --- The new parser's error includes the encoding name "BYTE_STREAM_SPLIT" --- without the old "E" prefix used in the previous error format. --- TODO: When BYTE_STREAM_SPLIT (encoding id=9) is implemented, change this --- to assertEqual checking actual dimensions. The encoding interleaves the --- individual byte streams of multi-byte scalars to improve compression for --- floating-point and other structured data: --- https://parquet.apache.org/docs/file-format/data-pages/encodings/#byte-stream-split-byte_stream_split--9 +-- TODO: Once BYTE_STREAM_SPLIT is implemented, assert actual dimensions. byteStreamSplitZstd :: Test byteStreamSplitZstd = TestCase @@ -1036,13 +892,7 @@ (D.readParquet "./tests/data/byte_stream_split.zstd.parquet") ) --- Was: assertExpectException "byteStreamSplitExtendedGzip" "FIXED_LEN_BYTE_ARRAY" ... --- The old parser had no page decoder for FIXED_LEN_BYTE_ARRAY and threw --- before ever inspecting the encoding. The new parser handles the physical --- type but the BYTE_STREAM_SPLIT encoding used for values is not yet --- implemented, so the error message shifts from the type to the encoding. --- TODO: Same as byteStreamSplitZstd — change to assertEqual once --- BYTE_STREAM_SPLIT encoding is supported. +-- TODO: Change to assertEqual once BYTE_STREAM_SPLIT is supported. byteStreamSplitExtendedGzip :: Test byteStreamSplitExtendedGzip = TestCase @@ -1052,18 +902,7 @@ (D.readParquet "./tests/data/byte_stream_split_extended.gzip.parquet") ) --- Was: assertExpectException "float16NonzerosAndNans" "PFIXED_LEN_BYTE_ARRAY" ... --- The "PFIXED_LEN_BYTE_ARRAY" in the old error was the Show of the old --- parser's ParquetType enum hitting a catch-all dispatch branch — it --- recognised the physical type but had no decoder for it. The new parser's --- fixedLenByteArrayDecoder reads 2-byte FIXED_LEN_BYTE_ARRAY (float16) --- columns as raw-byte text; proper float16 value decoding is not yet --- implemented. --- TODO: When IEEE 754 half-precision (float16) decoding is implemented, --- add a value-level assertion using hasElemType @Float (or a dedicated --- Float16 type if one is introduced). Verify that the decoded values match --- the known reference values for float16_nonzeros_and_nans.parquet. --- The column should no longer be exposed as raw-byte Text. +-- TODO: Once float16 decoding is implemented, assert the decoded values. float16NonzerosAndNans :: Test float16NonzerosAndNans = TestCase @@ -1078,11 +917,7 @@ ) ) --- Was: assertExpectException "float16ZerosAndNans" "PFIXED_LEN_BYTE_ARRAY" ... --- Same as float16NonzerosAndNans: old parser had no decoder for the --- FIXED_LEN_BYTE_ARRAY physical type; new parser reads raw bytes as text. --- TODO: Same as float16NonzerosAndNans — add a value-level assertion once --- float16 decoding is implemented. +-- TODO: Add a value-level assertion once float16 decoding is implemented. float16ZerosAndNans :: Test float16ZerosAndNans = TestCase @@ -1207,14 +1042,7 @@ ) ) --- Was: assertExpectException "unknownLogicalType" "Unknown logical type" ... --- The old parser raised a custom "Unknown logical type" message. The new --- Pinch-based metadata parser raises "Field 16 is absent" for the --- unrecognised LogicalType variant in this file. --- TODO: If Pinch is extended to support forward-compatible decoding of --- unknown union variants (treating unrecognised logical-type IDs as absent --- rather than raising an error), change this to assertEqual where the file --- parses successfully and the column falls back to its physical type. +-- TODO: If Pinch decodes unknown union variants, assert a successful parse. unknownLogicalType :: Test unknownLogicalType = TestCase @@ -1224,19 +1052,7 @@ (D.readParquet "./tests/data/unknown-logical-type.parquet") ) --- --------------------------------------------------------------------------- --- Group 12: Malformed files --- --------------------------------------------------------------------------- - --- Was: assertExpectException "nationDictMalformed" "dict index count mismatch" ... --- The old parser validated the dictionary entry count against data-page --- indices and raised "dict index count mismatch". The new parser does not --- replicate that check; the dictionary bytes happen to decode correctly --- despite the metadata discrepancy, returning the complete 25-row dataset. --- TODO: If a stricter dictionary-validation pass is added (checking that --- the number of decoded entries matches num_values in the dictionary page --- header), revert this to assertExpectException with a count-mismatch --- substring. +-- TODO: If stricter dictionary validation is added, expect a count mismatch. nationDictMalformed :: Test nationDictMalformed = TestCase @@ -1308,17 +1124,13 @@ "score should NOT be nullable" (not (hasMissing (unsafeGetColumn "score" df))) -{- | Nanosecond-precision timestamps (@TIMESTAMP(NANOS)@) must decode to the -correct 'UTCTime'. Regression test for the unit-scaling bug where the NANOS -multiplier @1_000_000 \`div\` 1_000_000_000@ truncated to 0, collapsing every -value to the epoch. --} +-- | Nanosecond timestamps must decode to the correct UTCTime. timestampNanos :: Test timestampNanos = testBothReadParquetPaths $ \readParquet -> TestCase $ assertEqual "TIMESTAMP(NANOS) decodes to correct UTCTime" - ( D.fromNamedColumns + ( D.fromColumns [ ( "ts" , D.fromList @@ -1394,6 +1206,7 @@ , -- Group 7: decimal types int32Decimal , int64Decimal + , nullableIntDecimal , byteArrayDecimal , fixedLengthDecimal , fixedLengthDecimalLegacy diff --git a/tests/ParquetTestData.hs b/tests/ParquetTestData.hs --- a/tests/ParquetTestData.hs +++ b/tests/ParquetTestData.hs @@ -1,9 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Large DataFrame fixtures used by the Parquet test suite. -Kept in a separate module to avoid cluttering Parquet.hs. --} +-- | Large DataFrame fixtures used by the Parquet test suite. module ParquetTestData ( allTypes, tinyPagesLast10, @@ -18,7 +16,7 @@ allTypes :: D.DataFrame allTypes = - D.fromNamedColumns + D.fromColumns [ ("id", D.fromList [4 :: Int32, 5, 6, 7, 2, 3, 0, 1]) , ("bool_col", D.fromList [True, False, True, False, True, False, True, False]) , ("tinyint_col", D.fromList [0 :: Int32, 1, 0, 1, 0, 1, 0, 1]) @@ -58,7 +56,7 @@ tinyPagesLast10 :: D.DataFrame tinyPagesLast10 = - D.fromNamedColumns + D.fromColumns [ ("id", D.fromList @Int32 (reverse [6174 .. 6183])) , ("bool_col", D.fromList @Bool (take 10 (cycle [False, True]))) , ("tinyint_col", D.fromList @Int32 [3, 2, 1, 0, 9, 8, 7, 6, 5, 4]) @@ -121,7 +119,7 @@ transactions :: D.DataFrame transactions = - D.fromNamedColumns + D.fromColumns [ ("transaction_id", D.fromList [1 :: Int32, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) , ( "event_time" @@ -279,7 +277,7 @@ mtCarsDataset :: D.DataFrame mtCarsDataset = - D.fromNamedColumns + D.fromColumns [ ( "model" , D.fromList diff --git a/tests/Plotting.hs b/tests/Plotting.hs --- a/tests/Plotting.hs +++ b/tests/Plotting.hs @@ -5,11 +5,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | -Tests for the Vega-Lite web plotting backend: field-type inference, the -box-plot mark fix, NaN handling, escaping, computed-expression encodings, and -typed/untyped spec parity. --} +-- | Tests for the Vega-Lite plotting backend, including typed parity. module Plotting (tests) where import Data.Aeson (Value (Array, Null, Object, String), toJSON) @@ -30,24 +26,19 @@ import qualified DataFrame.Display.Web.Chart.Typed as CT import qualified DataFrame.Display.Web.Plot as P --- Unqualified so the record update below has a single 'includeZero' field --- label in scope; GHC < 9.8 can't disambiguate the qualified duplicate field. +-- Unqualified: GHC < 9.8 can't disambiguate the qualified duplicate field. import DataFrame.Display.Web.Plot (Scatter (includeZero)) --- --------------------------------------------------------------------------- --- Fixtures + JSON helpers --- --------------------------------------------------------------------------- - numFrame :: D.DataFrame numFrame = - D.fromNamedColumns + D.fromColumns [ ("a", D.fromList ([1.0, 2.0, 3.0, 4.0] :: [Double])) , ("b", D.fromList ([10.0, 20.0, 30.0, 40.0] :: [Double])) ] mixedFrame :: D.DataFrame mixedFrame = - D.fromNamedColumns + D.fromColumns [ ("a", D.fromList ([1.0, 2.0, 3.0] :: [Double])) , ("g", D.fromList (["x", "y", "x"] :: [T.Text])) ] @@ -64,10 +55,6 @@ Just (Array xs) -> xs _ -> V.empty --- --------------------------------------------------------------------------- --- Test cases --- --------------------------------------------------------------------------- - fieldTypeInference :: Test fieldTypeInference = TestCase $ do let spec = @@ -107,7 +94,7 @@ nanBecomesNull :: Test nanBecomesNull = TestCase $ do - let df = D.fromNamedColumns [("a", D.fromList ([0 / 0, 1.0] :: [Double]))] + let df = D.fromColumns [("a", D.fromList ([0 / 0, 1.0] :: [Double]))] spec = C.toVegaSpec (C.chart df & C.enc C.Y (col @Double "a")) firstA = lookupKey "a" (fromMaybe Null (dataValues spec V.!? 0)) assertEqual "NaN inlines as null" (Just Null) firstA @@ -115,7 +102,7 @@ escapingSafe :: Test escapingSafe = TestCase $ do let weird = "we\"ir\\d" - df = D.fromNamedColumns [(weird, D.fromList ([1.0, 2.0] :: [Double]))] + df = D.fromColumns [(weird, D.fromList ([1.0, 2.0] :: [Double]))] spec = C.toVegaSpec (C.chart df & C.enc C.X (col @Double weird)) row0 = fromMaybe Null (dataValues spec V.!? 0) assertBool diff --git a/tests/PrettyPrint.hs b/tests/PrettyPrint.hs --- a/tests/PrettyPrint.hs +++ b/tests/PrettyPrint.hs @@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Golden tests pinning the format of 'prettyPrint'/'prettyPrintWidth': flat -@else if@ ladders, operator-precedence parenthesization, width-aware wrapping of -commutative chains, and short expressions staying on one line. --} +-- | Golden tests pinning the format of prettyPrint and prettyPrintWidth. module PrettyPrint (tests) where import qualified Data.Text as T @@ -42,6 +39,15 @@ (ifThenElse (col @Double "b" .>. lit 2.0) (lit "y") (lit "z")) ) ) + , golden + "caseWhen chain is a flat ladder" + "if a .>. 1.0\n \"x\"\nelse if b .>. 2.0\n \"y\"\nelse\n \"z\"" + ( prettyPrint + ( caseWhen (col @Double "a" .>. lit 1.0) (lit @T.Text "x") $ + caseWhen (col @Double "b" .>. lit 2.0) (lit "y") $ + orElse (lit "z") + ) + ) , -- A lower-precedence operand is parenthesized under a higher one. golden "precedence: (a + b) * c" "(a + b) * c" (prettyPrint ((a + b) * c)) , -- No spurious parens when precedence already disambiguates. @@ -63,7 +69,7 @@ "boolean chain fits: no paren noise" "a + b .>=. c .&&. a .>. b" (prettyPrintWidth 80 ((a + b .>=. c) .&&. (a .>. b))) - , -- When a nested operand wraps, parens delimit it; flat operands stay bare. + , -- When a nested operand wraps, parens delimit it; flat ones stay bare. golden "wrapped operand gets parens" "(a + b + c\n .>=. 0.0)\n .&&. a .>. b" diff --git a/tests/Properties/Categorical.hs b/tests/Properties/Categorical.hs --- a/tests/Properties/Categorical.hs +++ b/tests/Properties/Categorical.hs @@ -1,18 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | -Property tests pinning the categorical laws the library is meant to obey, -following https://mchav.github.io/what-category-theory-teaches-us-about-dataframes/ - - * /Topos (set algebra)/ — 'D.union', 'D.intersect', 'D.difference' and - 'D.symmetricDifference' form the subobject lattice, with 'D.distinct' - (image factorization) as the canonical set-valued map. - * /Migration functor Δ/ — 'D.select'/'D.rename'/'D.exclude' are functorial: - identities and round-trips hold. - -Operands that must share a schema are produced by row-subsetting one generated -base DataFrame, so the two/three frames are always schema-compatible. --} +-- | Property tests pinning the categorical laws of set algebra and migration. module Properties.Categorical (tests) where import Data.Text () @@ -30,11 +18,7 @@ nRows :: DataFrame -> Int nRows = fst . dataframeDimensions -{- | Equality of the underlying row sets, via the library's null-aware -@Eq DataFrame@. Set operations emit rows in a deterministic hash-bucket order -that depends only on content, and 'D.Eq' compares null slots correctly, so this -is a sound oracle for the topos laws (including over @Maybe Int@ columns). --} +-- | Equality of the underlying row sets, a sound oracle for the topos laws. sameRows :: DataFrame -> DataFrame -> Property sameRows x y = x === y @@ -44,20 +28,14 @@ -- | A schema-compatible triple, likewise. data Triple = Triple DataFrame DataFrame DataFrame deriving (Show) -{- | A base frame of @Int@ and @Maybe Int@ columns. - -@Maybe Int@ is included so the laws exercise nulls — @Nothing@ must behave as a -value distinct from any @Just n@. @Double@ is deliberately excluded: @NaN@/@-0.0@ -break set semantics by IEEE rules (@NaN /= NaN@), which is not a library bug. A -small value range makes duplicate rows common, exercising deduplication. --} +-- | A base frame of Int and Maybe Int columns, with many duplicate rows. genBase :: Gen DataFrame genBase = do nCols <- choose (1, 4) nRowsG <- choose (0, 60) let names = take nCols ["c0", "c1", "c2", "c3"] cols <- mapM (const (genCol nRowsG)) names - pure (D.fromNamedColumns (zip names cols)) + pure (D.fromColumns (zip names cols)) where genCol n = oneof @@ -87,12 +65,7 @@ base <- genBase Triple <$> subset base <*> subset base <*> subset base -{- | A single clean base frame (Int / Maybe Int columns only). - -Used by the Δ-functor laws, which compare with the representation-sensitive -@Eq DataFrame@ — and that @Eq@ is not even reflexive on @NaN@, so the shared -@Double@-bearing generator cannot be used here. --} +-- | A single clean base frame (Int and Maybe Int columns only). newtype Frame = Frame DataFrame deriving (Show) instance Arbitrary Frame where @@ -102,13 +75,6 @@ emptyLike :: DataFrame -> DataFrame emptyLike = D.range (0, 0) -------------------------------------------------------------------------------- --- Topos / set-algebra laws --- --- These are statements about row /sets/, so they are asserted with 'sameRows' --- (row-content equality) rather than the representation-sensitive @Eq DataFrame@. -------------------------------------------------------------------------------- - prop_unionCommutative :: Pair -> Property prop_unionCommutative (Pair a b) = sameRows (D.union a b) (D.union b a) @@ -151,10 +117,6 @@ prop_differenceDisjoint :: Pair -> Property prop_differenceDisjoint (Pair a b) = nRows (D.intersect (D.difference a b) b) === 0 - -------------------------------------------------------------------------------- --- Δ migration functor laws -------------------------------------------------------------------------------- -- | Excluding nothing is the identity. prop_excludeNothingIdentity :: Frame -> Property diff --git a/tests/Simplify.hs b/tests/Simplify.hs --- a/tests/Simplify.hs +++ b/tests/Simplify.hs @@ -2,9 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Specification for 'DataFrame.Internal.Expression.Simplify.simplify': each case is the -full predicate expression, compared with 'eqExpr'. --} +-- | Specification for simplify: each case compares the full expression. module Simplify (tests) where import DataFrame.Expression.Operators diff --git a/tests/Typed/IOReaders.hs b/tests/Typed/IOReaders.hs --- a/tests/Typed/IOReaders.hs +++ b/tests/Typed/IOReaders.hs @@ -2,11 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Tests for the typed CSV reader: a write\/read round-trip matches the untyped -reader, a wrong schema surfaces as 'Left' via 'readCsvWithError', and the -throwing 'readCsv' raises a 'DataFrameException'. The Parquet reader uses the -identical freeze-on-read path. --} +-- | Tests for the typed CSV reader: round-trip, wrong schema, throwing reads. module Typed.IOReaders (tests) where import Control.Exception (evaluate, try) @@ -32,7 +28,7 @@ sampleDF :: D.DataFrame sampleDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList [1, 2, 3 :: Int]) , ("y", DI.fromList [1.5, 2.5, 3.5 :: Double]) , ("g", DI.fromList ["a", "b", "c" :: T.Text]) @@ -62,9 +58,7 @@ IO (Either DataFrameException Int) assertBool "wrong schema => throws DataFrameException" (isLeft r) -{- | The schema is the read specification: only its columns are fetched, out -of a file that holds more. --} +-- | The schema is the read specification: only its columns are fetched. typedReadProjects :: Test typedReadProjects = TestCase $ withSystemTempFile "typed_proj.csv" $ \fp h -> do hClose h @@ -73,9 +67,7 @@ assertEqual "only the schema's column is read" ["g"] (D.columnNames narrow) assertEqual "rows intact" (3, 1) (D.dimensions narrow) -{- | The schema's types win over inference: a column of digits declared 'Text' -comes back as 'Text', which a plain read would have called 'Int'. --} +-- | The schema's types win over inference. typedReadSharesTypes :: Test typedReadSharesTypes = TestCase $ withSystemTempFile "typed_types.csv" $ \fp h -> do hClose h @@ -91,7 +83,7 @@ (Just (DI.fromList ["1", "2", "3" :: T.Text])) (getColumn "n" typed) where - digits = D.fromNamedColumns [("n", DI.fromList [1, 2, 3 :: Int])] + digits = D.fromColumns [("n", DI.fromList [1, 2, 3 :: Int])] tests :: [Test] tests = diff --git a/tests/Typed/Parity.hs b/tests/Typed/Parity.hs --- a/tests/Typed/Parity.hs +++ b/tests/Typed/Parity.hs @@ -25,7 +25,7 @@ baseDF :: D.DataFrame baseDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList [1, 2, 3, 4, 5, 6 :: Int]) , ("y", DI.fromList [10.0, 25.0, 30.0, 5.0, 50.0, 12.0 :: Double]) , ("g", DI.fromList ["a", "b", "a", "b", "a", "b" :: T.Text]) @@ -173,10 +173,10 @@ -- Disjoint frames for the (|||) test. leftDF :: D.DataFrame -leftDF = D.fromNamedColumns [("x", DI.fromList [1, 2 :: Int])] +leftDF = D.fromColumns [("x", DI.fromList [1, 2 :: Int])] rightDF :: D.DataFrame -rightDF = D.fromNamedColumns [("y", DI.fromList [1.0, 2.0 :: Double])] +rightDF = D.fromColumns [("y", DI.fromList [1.0, 2.0 :: Double])] leftT :: DT.TypedDataFrame '[ '("x", Int)] leftT = either (error . show) id (DT.freezeWithError leftDF) @@ -187,7 +187,7 @@ -- Numeric-only frame for the matrix test. numericDF :: D.DataFrame numericDF = - D.fromNamedColumns + D.fromColumns [ ("x", DI.fromList [1, 2, 3 :: Int]) , ("y", DI.fromList [1.5, 2.5, 3.5 :: Double]) ] diff --git a/tests/data/nullable_int_decimal.parquet b/tests/data/nullable_int_decimal.parquet new file mode 100644 Binary files /dev/null and b/tests/data/nullable_int_decimal.parquet differ