dataframe 3.6.1.0 → 3.7.0.0
raw patch · 80 files changed
+2098/−1372 lines, 80 filesdep ~dataframedep ~dataframe-coredep ~dataframe-expr-serializerbinary-added
Dependency ranges changed: dataframe, dataframe-core, dataframe-expr-serializer, dataframe-learn, dataframe-operations, dataframe-parquet
Files
- CHANGELOG.md +27/−1
- README.md +33/−47
- app/LazyBenchmark.hs +14/−40
- app/Main.hs +0/−2
- app/Synthesis.hs +2/−4
- benchmark/Main.hs +28/−41
- dataframe.cabal +35/−27
- src/DataFrame.hs +9/−23
- src/DataFrame/TH.hs +1/−16
- src/DataFrame/Typed.hs +3/−56
- src/DataFrame/Typed/TH.hs +1/−14
- tests/Assertions.hs +36/−1
- tests/Functions.hs +35/−2
- tests/IO/CSV.hs +2/−2
- tests/IO/CsvGolden.hs +3/−7
- tests/IO/JSON.hs +2/−2
- tests/IR/ExprJsonRoundtrip.hs +10/−16
- tests/Internal/ColumnBuilder.hs +1/−3
- tests/Internal/DictEncode.hs +2/−10
- tests/Internal/Markdown.hs +4/−7
- tests/Internal/PackedText.hs +58/−23
- tests/LazyParity.hs +7/−16
- tests/LazyParquet.hs +1/−1
- tests/LazyProjection.hs +5/−15
- tests/Learn/Denotation.hs +7/−13
- tests/Learn/Ensembles.hs +80/−17
- tests/Learn/Metamorphic.hs +14/−69
- tests/Learn/MetricsTests.hs +59/−3
- tests/Learn/Models.hs +12/−24
- tests/Learn/Segmented.hs +15/−14
- tests/Learn/SklearnParity.hs +1/−5
- tests/Learn/Synthesis.hs +5/−12
- tests/Learn/TypedModel.hs +5/−13
- tests/Main.hs +16/−0
- tests/Monad.hs +2/−2
- tests/Operations/Aggregations.hs +17/−17
- tests/Operations/Apply.hs +6/−6
- tests/Operations/Core.hs +5/−9
- tests/Operations/CountDistinct.hs +213/−0
- tests/Operations/Derive.hs +1/−1
- tests/Operations/ExprParse.hs +85/−0
- tests/Operations/Filter.hs +3/−3
- tests/Operations/GroupBy.hs +7/−15
- tests/Operations/Inference.hs +12/−31
- tests/Operations/InsertColumn.hs +6/−6
- tests/Operations/InterpreterKernels.hs +202/−0
- tests/Operations/Join.hs +28/−35
- tests/Operations/Merge.hs +12/−12
- tests/Operations/MonomorphicLoops.hs +324/−0
- tests/Operations/Nullable.hs +6/−30
- tests/Operations/NullableHashing.hs +8/−23
- tests/Operations/ParallelGroupBy.hs +45/−40
- tests/Operations/ParallelJoin.hs +12/−45
- tests/Operations/Provenance.hs +5/−17
- tests/Operations/ReadCsv.hs +14/−36
- tests/Operations/Record.hs +5/−5
- tests/Operations/RowWise.hs +148/−0
- tests/Operations/SetOps.hs +4/−6
- tests/Operations/Shuffle.hs +6/−27
- tests/Operations/Sort.hs +9/−9
- tests/Operations/SortMask.hs +102/−0
- tests/Operations/Statistics.hs +6/−10
- tests/Operations/Subset.hs +7/−8
- tests/Operations/Take.hs +1/−1
- tests/Operations/TextKernels.hs +81/−0
- tests/Operations/TopBy.hs +51/−0
- tests/Operations/Typing.hs +17/−24
- tests/Operations/VectorKernel.hs +8/−39
- tests/Operations/Window.hs +11/−12
- tests/Operations/WriteCsv.hs +6/−6
- tests/PackedTextMigration.hs +9/−26
- tests/Parquet.hs +44/−231
- tests/ParquetTestData.hs +5/−7
- tests/Plotting.hs +6/−19
- tests/PrettyPrint.hs +11/−5
- tests/Properties/Categorical.hs +5/−43
- tests/Simplify.hs +1/−3
- tests/Typed/IOReaders.hs +5/−13
- tests/Typed/Parity.hs +4/−4
- tests/data/nullable_int_decimal.parquet binary
CHANGELOG.md view
@@ -1,5 +1,31 @@ # Revision history for dataframe +## 3.7.0.0++* `dataframe-parquet`: a requested column is matched by its full name first,+ so a top-level column named with a dot (`sepal.length`) is read and projected+ as written; the last segment is only an alias for nested leaves. Fixes typed+ reads of such files, which failed with "Columns not found".+* `fromNamedColumns` is now `fromColumns`.+* `HasSchema`'s methods `toColumns`/`fromColumns` are now+ `recordsToColumns`/`recordsFromFrame`.+* `UExpr` is now `SomeExpr` (`toUExpr`/`fromUExpr` →+ `toSomeExpr`/`fromSomeExpr`).+* `dataframe-learn`: `GBConfig` gained fields (`gbMaxBins`, `gbL2`,+ `gbSubsample`, ...) and lost its `Eq` instance.+* `dataframe-parquet`: writer internals now run in `ST s` instead of `IO`.+* More ergonomic conditionals with `caseWhen`/`orElse`.+* `countDistinct`, `topBy`, and `(=:)` for building columns by hand.+* Tree learners handle nullable columns.+* Parallel parquet reader with streaming numeric decode.+* Faster parquet writer.+* Fused parallel kernels for whole-frame `sum`/`mean`, `countDistinct` and+ packed-text kernels.+* `pow` and `daysBetween` no longer treat their arguments as commutative.+* Nullable int32/int64 decimal parquet columns keep their nulls.+* The parquet reader selects and filters columns whose names contain dots.+* `countRows` reads files as UTF-8.+ ## 3.6.1.0 * splices for deriving schema@@ -119,7 +145,7 @@ A large performance and ML release. ### Highlights-* **Much faster I/O and analytics.** CSV reading, group-by, joins and sorting were rebuilt on compact unboxed / `PackedText` columns, parallel open-addressing hash tables, and vectorized aggregation; the default reader drops `cassava` for a single-pass scanner and `dataframe-fastcsv` adds multicore chunking. The end-to-end join + group-by pipeline runs several times faster with much lower memory and scales with `-threaded` / `+RTS -N`; results are byte-identical (golden-tested).+* **Much faster I/O and analytics.** CSV reading, group-by, joins and sorting were rebuilt on compact unboxed / `PackedText` columns, parallel open-addressing hash tables, and vectorized aggregation; the default reader drops `cassava` for a single-pass scanner and `dataframe-fastcsv` adds multicore chunking. The end-to-end join + group-by pipeline runs several times faster with much lower memory and scales with `-threaded` / `+RTS -N`. * **Lazy engine on par with eager** — bounded-source queries route through the same fast paths (streaming preserved for unbounded); lazy `sortBy` now also orders non-`Text` columns correctly. * **New ML library (`dataframe-learn`).** scikit-learn-style estimators behind a uniform `fit` / `predict`: linear / ridge / lasso / logistic regression, SVM, trees, boosting, k-means, GMM, DBSCAN, PCA and kernel PCA, symbolic regression and feature synthesis, with metrics and cross-validation. Pure and deterministic; every model also compiles to a dataframe `Expr`. * **Typed joins are checked at compile time** — keys must exist in both schemas with matching types (previously a runtime failure or silent empty result).
README.md view
@@ -35,19 +35,33 @@ # DataFrame -Tabular data analysis in Haskell. Read CSV, Parquet, and JSON files, transform columns with a typed expression DSL, and optionally lock down your entire schema at the type level for compile-time safety.+Tabular data analysis and symbolic AI in Haskell. Supports eager computation and lazy/deferred computation. API flexible enough to accomodate beginners to both data analysis or Haskell, and advanced type-safe pipeline. -The library ships three API layers — all operating on the same underlying `DataFrame` type at runtime:+## Features -- **Untyped** (`import qualified DataFrame as D`) — string-based column names, great for exploration and scripting.-- **Typed** (`import qualified DataFrame.Typed as T`) — phantom-type schema tracking with compile-time column validation.-- **Monadic API** — write your transformation as a self contained pipeline.+**I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from Hugging Face datasets (`hf://` URIs) via the `dataframe-huggingface` package. Column projection and predicate pushdown for Parquet reads. +**Operations**: filter, select, derive, groupBy, aggregate, joins (inner, left, right, full outer), sort, sample, stratified sample, distinct, k-fold splits.++**Expressions**: typed column references (`F.col @Double "x"`), arithmetic, comparisons, logical operators, nullable-aware three-valued logic (`.==`, `.&&`), string matching (`like`, `regex`), casting, and user-defined functions via `lift`/`lift2`.++**Statistics**: mean, median, mode, variance, standard deviation, percentiles, inter-quartile range, correlation, skewness, frequency tables, imputation.++**Plotting**: terminal plots (histogram, scatter, line, bar, box, pie, heatmap, stacked bar, correlation matrix) and interactive HTML plots.++**Lazy engine**: streaming query execution for files that don't fit in memory. Rule-based optimizer with filter fusion, predicate pushdown, and dead column elimination. Pull-based executor with configurable batch sizes.++**Interop**: Arrow C Data Interface for zero-copy round-trips with Python and Polars.++**ML**: decision trees (TAO algorithm), feature synthesis, k-fold cross-validation, stratified sampling.++**Notebooks**: IHaskell integration with [pre-built Binder examples](https://mybinder.org/v2/gh/mchav/ihaskell-dataframe/HEAD).+ > **This README is a runnable [scripths](https://github.com/DataHaskell/scripths) notebook.** Every Haskell block runs top-to-bottom in one shared session against the datasets in [`./data`](./data). Reproduce every output below with `scripths docs/base_scripts/base_readme.md -o README.md` from the repo root. ## Why this library? -* Concise, declarative, composable data pipelines using the `|>` pipe operator.+* Concise, declarative, composable data pipelines and machine learning. * Choose your level of type safety: keep it lightweight for quick analysis, or lock it down for production pipelines. * High performance from Haskell's optimizing compiler and an efficient columnar memory model based onn Apache Arrow. * Designed for interactivity: a custom REPL, IHaskell/Sabela notebook support, terminal and web plotting, and helpful error messages.@@ -69,18 +83,13 @@ ```bash $ dataframe-dataframe> df = D.fromNamedColumns [("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int]), ("amount", D.fromList [100, 120, 50, 20, 40, 30 :: Int]) ]+dataframe> df = D.fromColumns ["product" =: [1, 1, 2, 2, 3, 3 :: Int], "amount" =: [100, 120, 50, 20, 40, 30 :: Int]] dataframe> df |> D.groupBy ["product"] |> ["total" .= F.countAll ] ``` ## Quick Start -Group sales by product and compute totals. The first block carries the-`scripths` cabal directives; each later section imports what it needs where it-first uses it. You can also drop the same code into an `Example.hs` and run it-with `cabal run Example.hs` after adding a `#!/usr/bin/env cabal` header.- ```haskell -- cabal: build-depends: dataframe, text -- cabal: default-extensions: OverloadedStrings, TypeApplications, TemplateHaskell, DataKinds, TypeFamilies, FlexibleInstances, FlexibleContexts, ScopedTypeVariables, DeriveGeneric, UndecidableInstances@@ -88,9 +97,11 @@ import qualified DataFrame.Functions as F import DataFrame.Expression.Operators -sales = D.fromNamedColumns- [ ("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int])- , ("amount", D.fromList [100, 120, 50, 20, 40, 30 :: Int])+import DataFrame ((=:))++sales = D.fromColumns+ [ "product" =: [1, 1, 2, 2, 3, 3 :: Int]+ , "amount" =: [100, 120, 50, 20, 40, 30 :: Int] ] -- Group by product and compute totals@@ -278,11 +289,6 @@ ### Generate a schema (and a row bridge) from a record ADT -When the canonical row shape lives in your code as a Haskell record,-`deriveSchemaFromType` produces both the typed schema and a `HasSchema`-instance that converts between `[Order]` and a `DataFrame` (or-`TypedDataFrame OrderSchema`) at runtime:- ```haskell import Data.Text (Text) import Data.Int (Int64)@@ -299,8 +305,8 @@ -- '[ '("order_id", Int64), '("region", Text), '("amount", Double)] -- instance DT.HasSchema Order where -- type Schema Order = OrderSchema--- toColumns = ...--- fromColumns = ...+-- recordsToColumns = ...+-- recordsFromFrame = ... xs :: [Order] xs = [Order 1 "us" 10.0, Order 2 "eu" 20.5]@@ -389,8 +395,8 @@ instance DT.HasSchema OrderG where type Schema OrderG = OrderGSchema- toColumns = DT.genericToColumns- fromColumns = DT.genericFromColumns+ recordsToColumns = DT.genericToColumns+ recordsFromFrame = DT.genericFromColumns ``` > <!-- scripths:mime text/plain -->@@ -442,9 +448,9 @@ ```haskell type ScoreSchema = '[ '("name", Text), '("score", Maybe Double)] -scoresDf = D.fromNamedColumns- [ ("name", D.fromList ["a", "b", "c" :: Text])- , ("score", D.fromList [Just 1.0, Nothing, Just 3.0 :: Maybe Double])+scoresDf = D.fromColumns+ [ "name" =: ["a", "b", "c" :: Text]+ , "score" =: [Just 1.0, Nothing, Just 3.0 :: Maybe Double] ] Just stdf = DT.freeze @ScoreSchema scoresDf@@ -459,26 +465,6 @@ > | -------------|-----------------|----------------- | > | a | 1.0 | 100.0 | > | c | 3.0 | 300.0 |--## Features--**I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from Hugging Face datasets (`hf://` URIs) via the `dataframe-huggingface` package. Column projection and predicate pushdown for Parquet reads.--**Operations**: filter, select, derive, groupBy, aggregate, joins (inner, left, right, full outer), sort, sample, stratified sample, distinct, k-fold splits.--**Expressions**: typed column references (`F.col @Double "x"`), arithmetic, comparisons, logical operators, nullable-aware three-valued logic (`.==`, `.&&`), string matching (`like`, `regex`), casting, and user-defined functions via `lift`/`lift2`.--**Statistics**: mean, median, mode, variance, standard deviation, percentiles, inter-quartile range, correlation, skewness, frequency tables, imputation.--**Plotting**: terminal plots (histogram, scatter, line, bar, box, pie, heatmap, stacked bar, correlation matrix) and interactive HTML plots.--**Lazy engine**: streaming query execution for files that don't fit in memory. Rule-based optimizer with filter fusion, predicate pushdown, and dead column elimination. Pull-based executor with configurable batch sizes.--**Interop**: Arrow C Data Interface for zero-copy round-trips with Python and Polars.--**ML**: decision trees (TAO algorithm), feature synthesis, k-fold cross-validation, stratified sampling.--**Notebooks**: IHaskell integration with [pre-built Binder examples](https://mybinder.org/v2/gh/mchav/ihaskell-dataframe/HEAD). ## Lazy Queries
app/LazyBenchmark.hs view
@@ -2,15 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | End-to-end smoke test and benchmark for the Lazy streaming API:- generate a CSV then run five Lazy queries over it.-- Usage:- cabal run lazy-bench [-- [OPTIONS]] (+RTS -s -RTS for heap stats)- --rows N rows to generate (default 1_000_000_000)- --file PATH output CSV path (default: lazy_1b.csv in the system temp dir)- --skip-gen reuse the file if it already exists--}+-- | Benchmark for the Lazy API: generate a CSV, then run five Lazy queries. module Main where import Control.Monad (foldM, forM_, when)@@ -24,7 +16,7 @@ import DataFrame.Schema (Schema (..), schemaType) import System.Directory (doesFileExist, getFileSize, getTemporaryDirectory) import System.Environment (getArgs)-import System.Exit (exitFailure)+import System.Exit (exitFailure, exitSuccess) import System.IO ( BufferMode (..), IOMode (..),@@ -35,10 +27,6 @@ ) import System.Random.Stateful --- ------------------------------------------------------------------------------ Defaults--- ---------------------------------------------------------------------------- defaultRows :: Int defaultRows = 1_000_000_000 @@ -49,9 +37,15 @@ chunkSize :: Int chunkSize = 500_000 --- ------------------------------------------------------------------------------ Argument parsing--- ---------------------------------------------------------------------------+usage :: String+usage =+ unlines+ [ "Usage: cabal run lazy-bench [-- [OPTIONS]] (+RTS -s -RTS for heap stats)"+ , " --rows N rows to generate (default 1_000_000_000)"+ , " --file PATH output CSV path (default: lazy_1b.csv in the temp dir)"+ , " --skip-gen reuse the file if it already exists"+ , " --help show this message"+ ] data Opts = Opts { optRows :: Int@@ -70,16 +64,7 @@ go opts ("--skip-gen" : rest) = go opts{optSkipGen = True} rest go _ (flag : _) = Left $ "unknown flag: " ++ flag --- ------------------------------------------------------------------------------ CSV generation--- -----------------------------------------------------------------------------{- | Write @n@ rows of schema-- id (Int), x (Double), y (Double), category (Text: A/B/C/D)-- to @path@ using a streaming Builder, keeping heap usage constant.--}+-- | Stream @n@ rows of a four-column CSV to @path@ with constant heap. generateCsv :: FilePath -> Int -> IO () generateCsv path n = do g <- newIOGenM =<< newStdGen@@ -146,10 +131,6 @@ catChar 2 = Builder.char7 'C' catChar _ = Builder.char7 'D' --- ------------------------------------------------------------------------------ Lazy queries--- ---------------------------------------------------------------------------- runQuery :: String -> IO D.DataFrame -> IO () runQuery label action = do putStrLn $ "\n── " ++ label@@ -162,17 +143,14 @@ putStrLn $ " time : " ++ showDiff t0 t1 when (rows > 0 && rows <= 30) $ print df --- ------------------------------------------------------------------------------ Main--- ---------------------------------------------------------------------------- main :: IO () main = do hSetBuffering stdout LineBuffering args <- getArgs+ when ("--help" `elem` args) $ putStr usage >> exitSuccess defFile <- defaultFile opts <- case parseArgs defFile args of- Left err -> putStrLn ("Error: " ++ err) >> exitFailure+ Left err -> putStrLn ("Error: " ++ err) >> putStr usage >> exitFailure Right o -> return o let path = optFile opts@@ -244,10 +222,6 @@ $ L.scanCsv schema pathT putStrLn "\nDone."---- ------------------------------------------------------------------------------ Formatting helpers--- --------------------------------------------------------------------------- showDiff :: UTCTime -> UTCTime -> String showDiff t0 t1 = show (diffUTCTime t1 t0)
app/Main.hs view
@@ -34,10 +34,8 @@ cabalFile = cacheDir </> "df-repl.cabal" mainFile = cacheDir </> "Main.hs" - -- Ensure the directory is a valid Cabal project with a local target to repl into. ensureBoilerplate projectFile cabalFile mainFile - -- Keep your ghci script updated (weekly). shouldDownload <- needsUpdate ghciScript when shouldDownload $ do putStrLn "\ESC[92mDownloading latest version of dataframe config...\ESC[0m"
app/Synthesis.hs view
@@ -26,7 +26,7 @@ "./data/titanic/test.csv" ) --- Survived is Maybe Int (safeRead = MaybeRead); prediction is Int (model output).+-- Survived is Maybe Int (safeRead = MaybeRead); prediction is Int. type RawPredSchema = '[ '("Survived", Maybe Int), '("prediction", Int)] @@ -124,9 +124,7 @@ (w : _) -> T.dropEnd 1 w [] -> "" -{- | Compute binary classification accuracy from a DataFrame containing- "Survived" and "prediction" columns.--}+-- | Binary classification accuracy from "Survived" and "prediction" columns. computeAccuracy :: D.DataFrame -> Double computeAccuracy df = let tdf =
benchmark/Main.hs view
@@ -9,17 +9,13 @@ import Control.DeepSeq (NFData (..)) import Control.Monad (void) import Criterion.Main+import DataFrame ((=:)) import DataFrame.Expression.Operators import DataFrame.Internal.DataFrame (forceDataFrame) import DataFrame.Operations.Join import System.Process hiding (env) import System.Random.Stateful -{- | Criterion's 'nf' and 'env' force benchmark inputs/results to normal form-via 'NFData'. The core library intentionally dropped its @instance NFData-DataFrame@ in favour of 'forceDataFrame', so we provide a thin orphan here-(scoped to the benchmark) that reuses it.--} instance NFData D.DataFrame where rnf df = forceDataFrame df `seq` () @@ -107,10 +103,7 @@ parseMeasurementsTXT :: IO () parseMeasurementsTXT = parseFile "./data/measurements.txt" -{- | Generate a pair of dataframes for a 1:1 join scenario.- Left has keys [0..n-1], right has keys [0..n-1] shuffled.- overlap controls what fraction of keys appear in both sides.--}+-- | Two dataframes for a 1:1 join; overlap is the share of shared keys. mkOneToOne :: Int -> Double -> IO (D.DataFrame, D.DataFrame) mkOneToOne n overlap = do g <- newIOGenM =<< newStdGen@@ -119,23 +112,21 @@ let leftKeys = [0 :: Int .. n - 1] leftVals = [0 :: Int .. n - 1] leftDf =- D.fromNamedColumns- [ ("key", D.fromList leftKeys)- , ("A", D.fromList leftVals)+ D.fromColumns+ [ "key" =: leftKeys+ , "A" =: leftVals ] -- Right: take first `rightSize` keys, add non-overlapping keys for the rest rightPayload <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightSize] let rightKeys = [0 :: Int .. rightSize - 1] rightDf =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList rightKeys) , ("B", D.fromList rightPayload) ] return (leftDf, rightDf) -{- | Generate a pair of dataframes for a many-to-many join scenario.- Keys are drawn from [0..cardinality-1], so rows share keys.--}+-- | Two dataframes for a many-to-many join with keys in [0..cardinality-1]. mkManyToMany :: Int -> Int -> Int -> IO (D.DataFrame, D.DataFrame) mkManyToMany leftRows rightRows cardinality = do g <- newIOGenM =<< newStdGen@@ -145,22 +136,18 @@ leftVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. leftRows] rightVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightRows] let leftDf =- D.fromNamedColumns- [ ("key", D.fromList leftKeys)- , ("A", D.fromList leftVals)+ D.fromColumns+ [ "key" =: leftKeys+ , "A" =: leftVals ] rightDf =- D.fromNamedColumns- [ ("key", D.fromList rightKeys)- , ("B", D.fromList rightVals)+ D.fromColumns+ [ "key" =: rightKeys+ , "B" =: rightVals ] return (leftDf, rightDf) -{- | Generate a pair of dataframes for a many-to-one join- (fact table joining a dimension table).- Left has n rows with keys drawn from [0..dimSize-1].- Right has exactly dimSize rows with unique keys.--}+-- | Fact table and dimension table (unique keys) for a many-to-one join. mkManyToOne :: Int -> Int -> IO (D.DataFrame, D.DataFrame) mkManyToOne factRows dimSize = do g <- newIOGenM =<< newStdGen@@ -168,14 +155,14 @@ factVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. factRows] dimVals <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. dimSize] let factDf =- D.fromNamedColumns- [ ("key", D.fromList factKeys)- , ("A", D.fromList factVals)+ D.fromColumns+ [ "key" =: factKeys+ , "A" =: factVals ] dimDf =- D.fromNamedColumns- [ ("key", D.fromList [0 :: Int .. dimSize - 1])- , ("B", D.fromList dimVals)+ D.fromColumns+ [ "key" =: [0 :: Int .. dimSize - 1]+ , "B" =: dimVals ] return (factDf, dimDf) @@ -189,16 +176,16 @@ lv <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. leftRows] rv <- mapM (\_ -> uniformRM (0 :: Int, 1_000_000) g) [1 .. rightRows] let leftDf =- D.fromNamedColumns- [ ("key1", D.fromList lk1)- , ("key2", D.fromList lk2)- , ("A", D.fromList lv)+ D.fromColumns+ [ "key1" =: lk1+ , "key2" =: lk2+ , "A" =: lv ] rightDf =- D.fromNamedColumns- [ ("key1", D.fromList rk1)- , ("key2", D.fromList rk2)- , ("B", D.fromList rv)+ D.fromColumns+ [ "key1" =: rk1+ , "key2" =: rk2+ , "B" =: rv ] return (leftDf, rightDf)
dataframe.cabal view
@@ -1,6 +1,6 @@ cabal-version: 3.4 name: dataframe-version: 3.6.1.0+version: 3.7.0.0 synopsis: A fast, safe, and intuitive DataFrame library. description: A fast, safe, and intuitive DataFrame library for exploratory data analysis.@@ -127,13 +127,13 @@ DataFrame.Typed.Record, DataFrame.Typed.Generic build-depends: base >= 4 && <5,- dataframe-core >= 2.5 && < 2.6,+ dataframe-core >= 2.6 && < 2.7, dataframe-json >= 1.2.0.1 && < 1.3,- dataframe-expr-serializer >= 1.2.1 && < 1.3,- dataframe-operations >= 2.5 && < 2.6,+ dataframe-expr-serializer >= 1.3 && < 1.4,+ dataframe-operations >= 2.6 && < 2.7, dataframe-parsing >= 2.2 && < 2.3, dataframe-viz >= 1.3.2 && < 1.4,- dataframe-learn >= 2.4.2 && < 2.5+ dataframe-learn >= 2.5 && < 2.6 if !flag(no-csv) reexported-modules: DataFrame.IO.CSV,@@ -157,7 +157,7 @@ DataFrame.IO.Parquet.Time, DataFrame.IO.Utils.RandomAccess, DataFrame.Typed.IO.Parquet- build-depends: dataframe-parquet >= 1.5.0.1 && < 1.6+ build-depends: dataframe-parquet >= 1.6 && < 1.7 cpp-options: -DWITH_PARQUET -- The lazy executor calls both CSV and Parquet readers directly, so@@ -191,8 +191,8 @@ import: warnings main-is: Benchmark.hs build-depends: base >= 4 && < 5,- dataframe >= 3.6 && < 3.7,- dataframe-operations >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, time >= 1.12 && < 2, vector >= 0.13 && < 0.15,@@ -204,10 +204,10 @@ import: warnings main-is: Synthesis.hs build-depends: base >= 4 && < 5,- dataframe >= 3.6 && < 3.7,- dataframe-core >= 2.5 && < 2.6,- dataframe-learn >= 2.4.2 && < 2.5,- dataframe-operations >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-core >= 2.6 && < 2.7,+ dataframe-learn >= 2.5 && < 2.6,+ dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, text >= 2.1 && < 3 hs-source-dirs: app@@ -238,8 +238,8 @@ build-depends: base >= 4 && < 5, bytestring >= 0.11 && < 0.14, containers >= 0.6.7 && < 0.10,- dataframe >= 3.6 && < 3.7,- dataframe-core >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-core >= 2.6 && < 2.7, dataframe-lazy >= 2.4.1 && < 2.5, dataframe-parsing >= 2.2 && < 2.3, directory >= 1.3.0.0 && < 2,@@ -259,9 +259,9 @@ criterion >= 1 && < 2, deepseq >= 1.4 && < 2, process >= 1.6 && < 2,- dataframe >= 3.6 && < 3.7,- dataframe-core >= 2.5 && < 2.6,- dataframe-operations >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-core >= 2.6 && < 2.7,+ dataframe-operations >= 2.6 && < 2.7, random >= 1 && < 2, default-language: Haskell2010 ghc-options:@@ -298,12 +298,17 @@ Learn.SklearnParity, Learn.Synthesis, Learn.MetricsTests,+ Operations.ExprParse, Learn.Metamorphic, IO.CSV, IO.CsvGolden, IO.JSON, IR.ExprJsonRoundtrip, Operations.Aggregations,+ Operations.CountDistinct,+ Operations.SortMask,+ Operations.TextKernels,+ Operations.TopBy, Operations.Apply, Operations.Core, Operations.Derive,@@ -329,6 +334,9 @@ Operations.Take, Operations.Typing, Operations.VectorKernel,+ Operations.InterpreterKernels,+ Operations.MonomorphicLoops,+ Operations.RowWise, Operations.Record, LazyParquet, LazyParity,@@ -345,17 +353,17 @@ build-depends: base >= 4 && < 5, aeson >= 0.11.0.0 && < 3, bytestring >= 0.11 && < 0.14,- dataframe >= 3.6 && < 3.7,- dataframe-core >= 2.5 && < 2.6,- dataframe-core >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-core >= 2.6 && < 2.7,+ dataframe-core >= 2.6 && < 2.7, dataframe-csv >= 2.3 && < 2.4,- dataframe-expr-serializer >= 1.2.1 && < 1.3,+ dataframe-expr-serializer >= 1.3 && < 1.4, dataframe-fastcsv >= 1.4.0.1 && < 1.5, dataframe-json >= 1.2.0.1 && < 1.3, dataframe-lazy >= 2.4.1 && < 2.5,- dataframe-learn >= 2.4.2 && < 2.5,- dataframe-operations >= 2.5 && < 2.6,- dataframe-parquet >= 1.5.0.1 && < 1.6,+ dataframe-learn >= 2.5 && < 2.6,+ dataframe-operations >= 2.6 && < 2.7,+ dataframe-parquet >= 1.6 && < 1.7, dataframe-parsing >= 2.2 && < 2.3, HUnit >= 1.6 && < 1.8, QuickCheck >= 2 && < 3,@@ -381,9 +389,9 @@ other-modules: Internal.PackedText build-depends: base >= 4 && < 5, bytestring >= 0.11 && < 0.14,- dataframe >= 3.6 && < 3.7,- dataframe-core >= 2.5 && < 2.6,- dataframe-operations >= 2.5 && < 2.6,+ dataframe >= 3.7 && < 3.8,+ dataframe-core >= 2.6 && < 2.7,+ dataframe-operations >= 2.6 && < 2.7, HUnit >= 1.6 && < 1.8, text >= 2.1 && < 3, vector >= 0.13 && < 0.15
src/DataFrame.hs view
@@ -1,17 +1,6 @@ {-# LANGUAGE CPP #-} -{- |-Module : DataFrame-Copyright : (c) 2024 - 2026 Michael Chavinda-License : GPL-3.0-Maintainer : mschavinda@gmail.com-Stability : experimental-Portability : POSIX--Batteries-included entry point for @dataframe@: re-exports the most commonly-used pieces for GHCi and scripts. Use the @D.@ prefix for core table operations-and @F.@ for the expression DSL.--}+-- | Batteries-included entry point re-exporting the common pieces for GHCi. module DataFrame ( -- * Core data structures module CoreTypes,@@ -79,9 +68,7 @@ ) where --- DecisionTree's own `percentile`/`percentiles` are hidden here; the public--- versions come from Operations.Statistics / DataFrame.Synthesis, avoiding--- ambiguous-export errors.+-- DecisionTree's `percentile`/`percentiles` are hidden to avoid ambiguity. import DataFrame.DecisionTree as DecisionTree import DataFrame.Display as Display ( DisplayOptions (..),@@ -111,8 +98,7 @@ readJSON, readJSONEither, )--- Marks the dataframe-expr-serializer dependency as used (the codec modules are--- surfaced to consumers via this library's @reexported-modules@).+-- Marks the dataframe-expr-serializer dependency as used. import DataFrame.Expr.Serialize () #ifdef WITH_PARQUET import DataFrame.IO.Parquet as Parquet (@@ -144,13 +130,13 @@ columnNames, defaultTruncateConfig, toNamedExpr,- toUExpr,- fromUExpr,+ toSomeExpr,+ fromSomeExpr, eSize, empty, fromAny, fromList,- fromNamedColumns,+ fromColumns, fromUnboxedVector, fromVector, hasElemType,@@ -172,6 +158,7 @@ toRowVector, toSeparated, toVector,+ (=:), ) import DataFrame.Schema as Schema ( SchemaType (..),@@ -182,9 +169,7 @@ import DataFrame.Typed.TH.Records as SchemaTH (deriveSchemaFromType, deriveSchemaValues) #endif #ifdef WITH_LAZY--- Re-export only the lazy engine's types and source constructors; its--- operator surface (filter/select/derive/...) collides with the eager API,--- so users wanting full lazy access `import qualified DataFrame.Lazy as L`.+-- Re-export only the lazy types and sources; its operators clash with eager. import DataFrame.Lazy as Lazy ( LazyDataFrame, fromDataFrame,@@ -221,6 +206,7 @@ SortOrder (..), shuffle, sortBy,+ topBy, ) import DataFrame.Operations.Statistics as Statistics ( correlation,
src/DataFrame/TH.hs view
@@ -1,21 +1,6 @@ {-# LANGUAGE CPP #-} -{- |-Module : DataFrame.TH-License : MIT--Backwards-compatibility re-export hub for the split @DataFrame.TH.*@-modules. New code can import the specific submodule directly:-- * "DataFrame.TH.Records" — record/DataFrame-based splices (no file IO)- * "DataFrame.TH.CSV" — CSV-file-based splices (requires @-fwith-csv@)- * "DataFrame.TH.Parquet" — Parquet-file-based splices (requires @-fwith-parquet@)--These live in @dataframe-th@, @dataframe-csv-th@, and-@dataframe-parquet-th@ respectively. The CSV/Parquet re-exports are-guarded with the @CSV_TH@ / @PARQUET_TH@ CPP defines that the-meta-@dataframe@ package sets based on its cabal flags.--}+-- | Backwards-compatibility re-export hub for the split TH modules. module DataFrame.TH ( module DataFrame.TH.Records, #ifdef WITH_CSV_TH
src/DataFrame/Typed.hs view
@@ -1,62 +1,7 @@ {-# LANGUAGE CPP #-} {-# LANGUAGE DataKinds #-} -{- |-Module : DataFrame.Typed-Copyright : (c) 2024 - 2026 Michael Chavinda-License : MIT-Maintainer : mschavinda@gmail.com-Stability : experimental--A type-safe layer over the @dataframe@ library.--This module provides 'TypedDataFrame', a phantom-typed wrapper around-the untyped 'DataFrame' that tracks column names and types at compile time.-All operations delegate to the untyped core at runtime; the phantom type-is updated at compile time to reflect schema changes.--== Key difference from untyped API: TExpr--All expression-taking operations use 'TExpr' (typed expressions) instead-of raw @Expr@. Column references are validated at compile time:--@-{\-\# LANGUAGE DataKinds, TypeApplications, TypeOperators \#-\}-import qualified DataFrame.Typed as T--type People = '[ '(\"name\", Text), '(\"age\", Int)]--main = do- raw <- D.readCsv \"people.csv\"- case T.freeze \@People raw of- Nothing -> putStrLn \"Schema mismatch!\"- Just df -> do- let adults = T.filterWhere (T.col \@\"age\" T..>=. T.lit 18) df- let names = T.columnAsList \@\"name\" adults -- :: [Text]- print names-@--Column references like @T.col \@\"age\"@ are checked at compile time — if the-column doesn't exist or has the wrong type, you get a type error, not a-runtime exception.--== filterAllJust tracks Maybe-stripping--@-df :: TypedDataFrame '[ '(\"x\", Maybe Double), '(\"y\", Int)]-T.filterAllJust df :: TypedDataFrame '[ '(\"x\", Double), '(\"y\", Int)]-@--== Typed aggregation--@-result = T.aggregate- ( T.as \@\"total\" (T.sum (T.col \@\"salary\"))- . T.as \@\"count\" (T.count (T.col \@\"salary\"))- )- (T.groupBy \@'[\"dept\"] employees)-@--}+-- | A type-safe layer that tracks column names and types over the core API. module DataFrame.Typed ( -- * Core types TypedDataFrame,@@ -68,6 +13,8 @@ col, lit, ifThenElse,+ caseWhen,+ orElse, lift, lift2, nullLift,
src/DataFrame/Typed/TH.hs view
@@ -1,19 +1,6 @@ {-# LANGUAGE CPP #-} -{- |-Module : DataFrame.Typed.TH-License : MIT--Backwards-compatibility re-export hub for the split @DataFrame.Typed.TH.*@-modules:-- * "DataFrame.Typed.TH.Records" — record-based schema derivation (no file IO)- * "DataFrame.Typed.TH.CSV" — CSV-file-based schema derivation (requires @-fwith-csv@)--These live in @dataframe-th@ and @dataframe-csv-th@ respectively. The CSV-re-export is guarded with the @CSV_TH@ CPP define that the-meta-@dataframe@ package sets based on its cabal flags.--}+-- | Backwards-compatibility re-export hub for @DataFrame.Typed.TH.*@ modules. module DataFrame.Typed.TH ( module DataFrame.Typed.TH.Records, #ifdef WITH_CSV_TH
tests/Assertions.hs view
@@ -5,9 +5,10 @@ import qualified Data.List as L import Control.Exception+import GHC.Float (castDoubleToWord64, castFloatToWord32) import Test.HUnit --- Adapted from: https://github.com/BartMassey/chunk/blob/1ee4bd6545e0db6b8b5f4935d97e7606708eacc9/hunit.hs#L29+-- Adapted from BartMassey/chunk, hunit.hs. assertExpectException :: String -> String ->@@ -33,3 +34,37 @@ ++ expected ++ ", got: " ++ show e++-- | Equality that tells NaNs and signed zeros apart for floating types.+class BitEq a where+ bitEq :: a -> a -> Bool++instance BitEq Double where+ bitEq a b = castDoubleToWord64 a == castDoubleToWord64 b+instance BitEq Float where+ bitEq a b = castFloatToWord32 a == castFloatToWord32 b+instance BitEq Int where+ bitEq = (==)+instance BitEq Bool where+ bitEq = (==)+instance (BitEq a) => BitEq (Maybe a) where+ bitEq (Just a) (Just b) = bitEq a b+ bitEq Nothing Nothing = True+ bitEq _ _ = False++-- | Lists equal under 'bitEq'; on failure, reports the first differing row.+assertBitEq :: (BitEq a, Show a) => String -> [a] -> [a] -> Assertion+assertBitEq label expected actual+ | length expected /= length actual =+ assertFailure+ ( label+ ++ ": "+ ++ show (length actual)+ ++ " rows, expected "+ ++ show (length expected)+ )+ | otherwise = case [(i, e, a) | (i, e, a) <- zip3 [0 :: Int ..] expected actual, not (bitEq e a)] of+ [] -> pure ()+ (i, e, a) : _ ->+ assertFailure+ (label ++ ": row " ++ show i ++ " expected " ++ show e ++ ", got " ++ show a)
tests/Functions.hs view
@@ -3,6 +3,7 @@ module Functions where +import Data.Time.Calendar (Day, fromGregorian) import qualified DataFrame as D import DataFrame.Functions ( sanitize,@@ -63,7 +64,7 @@ ] df :: D.DataFrame df =- D.fromNamedColumns+ D.fromColumns [("A", DI.fromList [(1 :: Int) .. 10])] testSum :: Test@@ -71,7 +72,7 @@ TestCase ( assertEqual "Sum first 10 numbers"- ( D.fromNamedColumns+ ( D.fromColumns [ ("A", DI.fromList [(1 :: Int) .. 10]) , ("sum", DI.fromList (replicate 10 (55 :: Int))) ]@@ -79,8 +80,40 @@ (D.derive "sum" (F.sum (F.col @Int "A")) df) ) +testPow :: Test+testPow =+ TestCase+ ( assertEqual+ "pow of a compound base"+ [4, 9, 16, 25, 36, 49, 64, 81, 100, 121]+ ( D.columnAsList @Int+ (F.col "sq")+ (D.derive "sq" (F.pow (F.col @Int "A" + F.lit 1) 2) df)+ )+ )++testDaysBetween :: Test+testDaysBetween =+ TestCase+ ( assertEqual+ "daysBetween d1 d2 is d1 minus d2 in either argument order"+ ([-9], [9])+ ( days (F.col @Day "start") (F.col @Day "end")+ , days (F.col @Day "end") (F.col @Day "start")+ )+ )+ where+ dates =+ D.fromColumns+ [ ("start", DI.fromList [fromGregorian 2024 3 1])+ , ("end", DI.fromList [fromGregorian 2024 3 10])+ ]+ days a b = D.columnAsList @Int (F.col "d") (D.derive "d" (F.daysBetween a b) dates)+ tests :: [Test] tests = [ TestLabel "sanitizeIdentifiers" sanitizeIdentifiers , TestLabel "testSum" testSum+ , TestLabel "testPow" testPow+ , TestLabel "testDaysBetween" testDaysBetween ]
tests/IO/CSV.hs view
@@ -46,7 +46,7 @@ fromCsvRoundTrip :: Test fromCsvRoundTrip = TestLabel "fromCsv_roundTrip" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList @Int [1, 2, 3]) , ("b", DI.fromList @T.Text ["hello", "world", "test"]) ]@@ -65,7 +65,7 @@ fromCsvBytesRoundTrip :: Test fromCsvBytesRoundTrip = TestLabel "fromCsvBytes_roundTrip" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList @Int [10, 20]) , ("y", DI.fromList @Double [1.5, 2.5]) ]
tests/IO/CsvGolden.hs view
@@ -2,10 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | Golden semantics for the default CSV reader, pinned against the-cassava-based oracle (2026-06-12) before the strict-scanner rewrite.-Ragged-row cases encode the one intentional change: pad-with-null (audit D6).--}+-- | Golden semantics for the default CSV reader, pinned against the old oracle. module IO.CsvGolden (tests) where import qualified Data.ByteString as BS@@ -586,8 +583,7 @@ assertEqual "dims" (2, 2) (dataframeDimensions df) assertEqual "index map" [("a", 1)] (M.toList (columnIndices df)) --- Ingest pin: a string column freezes to a shared-buffer 'PackedText'--- (no per-row boxed Text header), while values stay byte-identical.+-- Ingest pin: string columns freeze to a shared-buffer 'PackedText'. textFreezesPacked :: Test textFreezesPacked = TestLabel "text_freezes_packed" $ TestCase $ do df <- decodeSeparated defaultReadOptions (BL.fromStrict "a,b\n1,x\n2,y\n")@@ -595,7 +591,7 @@ Nothing -> assertFailure "missing text column" Just col -> do assertBool "text column is PackedText" (DI.isPackedText col)- assertEqual "values byte-identical" (texts ["x", "y"]) col+ assertEqual "values equal" (texts ["x", "y"]) col tests :: [Test] tests = dupHeaders : textFreezesPacked : map goldenCase goldenCases
tests/IO/JSON.hs view
@@ -46,13 +46,13 @@ Right df -> do assertEqual "Missing key: 2 rows" 2 (D.nRows df) assertEqual "Missing key: 2 columns" 2 (D.nColumns df)- -- 'b' is absent from the second row, so must have a missing value+ -- 'b' is absent from the second row, so it is missing assertBool "b column should have missing values" (maybe False DI.hasMissing (DI.getColumn "b" df)) ) --- | When column values are different types across rows, the column becomes generic.+-- | Column values of different types across rows make the column generic. jsonMixedTypeColumn :: Test jsonMixedTypeColumn = TestCase
tests/IR/ExprJsonRoundtrip.hs view
@@ -3,9 +3,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Round-trip, pipeline, end-to-end, and wire-shape contract tests for the-expression/pipeline serializer.--}+-- | Round-trip and wire-shape contract tests for the expression serializer. module IR.ExprJsonRoundtrip (tests) where import qualified Data.Aeson as Aeson@@ -19,7 +17,7 @@ import qualified DataFrame.Functions as F import DataFrame.Internal.Column (Columnable, TypedColumn (..), toVector) import qualified DataFrame.Internal.Column as DI-import DataFrame.Internal.Expression (Expr, UExpr (..))+import DataFrame.Internal.Expression (Expr, SomeExpr (..)) import DataFrame.Internal.Interpreter (interpret) import DataFrame.LinearModel (defaultLinearConfig)@@ -31,7 +29,6 @@ ) import DataFrame.Expr.Serialize (- SomeExpr (..), decodeExprAny, decodeExprAt, decodeExprFromBytes,@@ -52,11 +49,11 @@ close a b = length a == length b && and (zipWith (\x y -> abs (x - y) <= 1e-9) a b) df3 :: D.DataFrame-df3 = D.fromNamedColumns [("x", DI.fromList [1, 2, 3 :: Double])]+df3 = D.fromColumns [("x", DI.fromList [1, 2, 3 :: Double])] regDF :: D.DataFrame regDF =- D.fromNamedColumns+ D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2])@@ -65,13 +62,11 @@ xs1 = [1, 2, 3, 4, 5, 6, 7, 8] :: [Double] xs2 = [2, 1, 4, 3, 6, 5, 8, 7] :: [Double] -{- | An expression round-trips when encode → decode → re-encode yields the same-JSON. Avoids needing an 'Eq' instance on the GADT.--}+-- | Round-trips when encode, decode, re-encode yields the same JSON. roundtrips :: (Columnable a) => Expr a -> Bool roundtrips e = fromRight False $ do v1 <- encodeExpr e- SomeExpr _ e' <- decodeExprAny v1+ SomeExpr e' <- decodeExprAny v1 v2 <- encodeExpr e' Right (v1 == v2) @@ -109,8 +104,7 @@ , testCase "over" (roundtrips (F.over ["g"] (F.sum (F.col @Double "x")))) ] --- Documented limitation: collect's list-typed output has no type tag, so it--- cannot be encoded.+-- Documented limitation: collect's list-typed output cannot be encoded. limitationTests :: [Test] limitationTests = [ testCase@@ -161,8 +155,8 @@ , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do let fp = dir ++ "/pipeline.json" nes =- [ ("z", UExpr (F.col @Double "x" * F.lit 2))- , ("w", UExpr (F.col @Double "x" + F.lit 10))+ [ ("z", SomeExpr (F.col @Double "x" * F.lit 2))+ , ("w", SomeExpr (F.col @Double "x" + F.lit 10)) ] sr <- savePipelineToFile fp nes assertEqual "save pipeline ok" (Right ()) sr@@ -175,7 +169,7 @@ assertBool "w = x+10" (close (interpD df' (F.col @Double "w")) [11, 12, 13]) , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do let fp = dir ++ "/transform.json"- t = Transform [("z", UExpr (F.col @Double "x" * F.lit 2))]+ t = Transform [("z", SomeExpr (F.col @Double "x" * F.lit 2))] sr <- saveTransformToFile fp t assertEqual "save transform ok" (Right ()) sr loaded <- loadTransformFromFile fp
tests/Internal/ColumnBuilder.hs view
@@ -100,7 +100,6 @@ | otherwise = fromVector (VB.fromList (catMaybes xs)) -- Read a text column back as a Maybe list, respecting the bitmap.--- Ingest now freezes text to 'PackedText'; materialize to the boxed form. columnTexts :: Column -> [Maybe T.Text] columnTexts c@(PackedText _ _) = columnTexts (materializePacked c) columnTexts (BoxedColumn mb (v :: VB.Vector b)) =@@ -191,8 +190,7 @@ (map Just ts) (columnTexts (buildTextColumn (map Just ts))) --- A multibyte sequence split across two fields: the whole buffer is valid--- UTF-8 but each field alone is not. Both must lenient-decode to U+FFFD.+-- A multibyte sequence split across fields must lenient-decode to U+FFFD. splitMultibyteAcrossFields :: Test splitMultibyteAcrossFields = TestCase $ do let fields = [[0xC2], [0x80]]
tests/Internal/DictEncode.hs view
@@ -1,13 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ScopedTypeVariables #-} -{- | Correctness oracle for the dictionary-encode building block-('DataFrame.Internal.Column.Encode'). A text column encodes to dense-first-appearance @Int@ codes: two rows share a code iff their text is equal, the-codes are a contiguous @0 .. card-1@ range in first-appearance order, packed and-boxed Text encode identically, and the cap parameter bails past a cardinality.-Pins the step that the group-by planner can route a text key through.--}+-- | Correctness oracle for dictionary-encoding text columns into dense codes. module Internal.DictEncode (tests) where import qualified Data.ByteString as B@@ -56,9 +50,7 @@ boxedFromTexts :: [T.Text] -> DI.Column boxedFromTexts = DI.fromList -{- | The reference dense first-appearance encoding computed with a plain-'Data.Map': scan in row order, assign the next id to each new value.--}+-- | The reference first-appearance encoding, computed with a plain Map. oracleCodes :: [T.Text] -> ([Int], Int) oracleCodes ts = go ts M.empty 0 [] where
tests/Internal/Markdown.hs view
@@ -1,8 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | Markdown rendering must escape pipe-table metacharacters in cell values,-or an operator name like @<|>@ splits the row into the wrong columns.--}+-- | Markdown rendering must escape pipe-table metacharacters in cell values. module Internal.Markdown (tests) where import qualified Data.Text as T@@ -12,8 +10,7 @@ import qualified DataFrame.Internal.DataFrame as D import Test.HUnit --- Count the column-delimiter pipes in a rendered row: a bare '|' delimits a--- cell; an escaped "\|" does not.+-- Count a rendered row's delimiter pipes; an escaped "\|" is not one. unescapedPipes :: T.Text -> Int unescapedPipes = go . T.unpack where@@ -31,7 +28,7 @@ , TestLabel "toMarkdown keeps an operator name in one cell" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("name", fromList ["plain" :: T.Text, "f<|>g"]) , ("mb", fromList [1.0 :: Double, 2.0]) ]@@ -45,7 +42,7 @@ , TestLabel "nullable boxed Text renders without quotes" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [("name", ensureOptional (fromList ["Ada" :: T.Text]))] md = D.toMarkdown df assertBool
tests/Internal/PackedText.hs view
@@ -2,12 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | Correctness oracle for the packed-text column variant. Every assertion-builds a 'PackedText' column from raw bytes + offsets and the same data as a-'BoxedColumn Text', then checks display, extraction, equality, ordering,-hashing, groupBy and join all agree. This pins behavioral parity before any-reader emits packed columns.--}+-- | Correctness oracle: packed-text columns behave like boxed Text columns. module Internal.PackedText (tests) where import qualified Data.Text as T@@ -20,6 +15,7 @@ import Data.Text.Encoding (encodeUtf8) import Data.Word (Word8) import qualified DataFrame as D+import DataFrame.Expression.Operators ((./=.), (.==.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI import DataFrame.Internal.Data.PackedText (mkPackedContiguous)@@ -67,8 +63,8 @@ -- Single-column dataframes for groupBy / hashing. packedDf, boxedDf :: D.DataFrame-packedDf = D.fromNamedColumns [("k", packedCol)]-boxedDf = D.fromNamedColumns [("k", boxedCol)]+packedDf = D.fromColumns [("k", packedCol)]+boxedDf = D.fromColumns [("k", boxedCol)] displayParity :: Test displayParity =@@ -116,15 +112,14 @@ joinParity :: Test joinParity = TestCase $ do- let lhsP = D.fromNamedColumns [("k", packedCol)]- lhsB = D.fromNamedColumns [("k", boxedCol)]- rhs = D.fromNamedColumns [("k", boxedFromTexts ["apple", "banana", "日本語"])]+ let lhsP = D.fromColumns [("k", packedCol)]+ lhsB = D.fromColumns [("k", boxedCol)]+ rhs = D.fromColumns [("k", boxedFromTexts ["apple", "banana", "日本語"])] joinedP = D.innerJoin ["k"] lhsP rhs joinedB = D.innerJoin ["k"] lhsB rhs assertBool "inner join parity" (joinedP == joinedB) --- A gather (atIndicesStable) on a packed column stays packed and equals the--- boxed gather. Indices repeat, reorder, and drop rows.+-- A gather on a packed column stays packed and equals the boxed gather. gatherPreservesPacked :: Test gatherPreservesPacked = TestCase $ do let ixs = VU.fromList [9, 0, 0, 5, 7, 2, 3 :: Int]@@ -147,19 +142,18 @@ (DI.isPackedText (unsafeGetColumn "k" sortedP)) assertBool "sorted packed == sorted boxed" (sortedP == sortedB) --- A left join (sentinel gather, -1 for missing right rows) on a packed right--- column stays packed, builds the correct null bitmap, and equals boxed.+-- A left join on a packed right column stays packed and equals boxed. leftJoinSentinelPreservesPacked :: Test leftJoinSentinelPreservesPacked = TestCase $ do- let lhsP = D.fromNamedColumns [("k", packedCol)]- lhsB = D.fromNamedColumns [("k", boxedCol)]+ let lhsP = D.fromColumns [("k", packedCol)]+ lhsB = D.fromColumns [("k", boxedCol)] rhsP =- D.fromNamedColumns+ D.fromColumns [ ("k", packedFromTexts ["apple", "banana", "日本語"]) , ("v", packedFromTexts ["RA", "RB", "RC"]) ] rhsB =- D.fromNamedColumns+ D.fromColumns [ ("k", boxedFromTexts ["apple", "banana", "日本語"]) , ("v", boxedFromTexts ["RA", "RB", "RC"]) ]@@ -170,8 +164,7 @@ (DI.isPackedText (unsafeGetColumn "v" joinedP)) assertBool "left join packed == boxed" (joinedP == joinedB) --- A slice of a packed column stays packed: decoding to Text would make a--- ten-row slice cost the whole frame.+-- A slice of a packed column stays packed instead of decoding to Text. slicePreservesPacked :: Test slicePreservesPacked = TestCase $ do let sp = DI.sliceColumn 3 4 packedCol@@ -194,8 +187,7 @@ (DI.toList @T.Text tb) (DI.toList @T.Text tp) --- Every other representation rejects a slice running past the end; the packed--- arm must not quietly decode the overrun as empty strings.+-- A packed slice past the end must be rejected, not decoded as empty. sliceRejectsInvalidBounds :: Test sliceRejectsInvalidBounds = TestCase $ do r <- try (evaluate (DI.columnLength (DI.sliceColumn 8 5 packedCol)))@@ -204,6 +196,48 @@ Right len -> assertFailure ("expected an invalid slice, got length " ++ show len) +-- Rows for the eq/neq kernel: valid text, empty, and invalid UTF-8.+kernelRowBytes :: [[Word8]]+kernelRowBytes =+ map (B.unpack . encodeUtf8) ["apple", "", "café", "日本語", "apple", "\xFFFD"]+ ++ [[0xFF], [0x63, 0x61, 0x66, 0xC3], [0x80], [0xEF, 0xBF]]++packedFromBytes :: [[Word8]] -> DI.Column+packedFromBytes rows =+ DI.PackedText+ Nothing+ ( mkPackedContiguous+ (arrayFromBytes (concat rows))+ (VU.fromList (scanl (+) 0 (map length rows)))+ )++-- The eq/neq-literal kernel must agree with the decoded-column expression.+eqLiteralKernelParity :: Test+eqLiteralKernelParity = TestCase $ do+ let base = D.fromColumns [("k", packedFromBytes kernelRowBytes)]+ sorted = D.sortBy [D.Asc (F.col @T.Text "k")] base+ boxed df = D.fromColumns [("k", DI.materializePacked (unsafeGetColumn "k" df))]+ lits = ["apple", "", "café", "日本語", "\xFFFD", "caf", "missing"] :: [T.Text]+ result :: D.DataFrame -> D.Expr Bool -> [Bool]+ result df e = DI.toList @Bool (unsafeGetColumn "r" (D.derive "r" e df))+ assertBool+ "sorted payload stays PackedText"+ (DI.isPackedText (unsafeGetColumn "k" sorted))+ sequence_+ [ assertEqual+ (label ++ " " ++ show lit)+ (result (boxed df) e)+ (result df e)+ | (label, df) <- [("base", base), ("sorted", sorted)]+ , lit <- lits+ , e <-+ [ F.col @T.Text "k" .==. F.lit lit+ , F.col @T.Text "k" ./=. F.lit lit+ , F.lit lit .==. F.col @T.Text "k"+ , F.lit lit ./=. F.col @T.Text "k"+ ]+ ]+ tests :: [Test] tests = [ TestLabel "PackedText display parity" displayParity@@ -221,4 +255,5 @@ , TestLabel "PackedText slice preserves packed" slicePreservesPacked , TestLabel "PackedText takeLast preserves packed" takeLastPreservesPacked , TestLabel "PackedText slice rejects bad bounds" sliceRejectsInvalidBounds+ , TestLabel "PackedText eq/neq literal kernel parity" eqLiteralKernelParity ]
tests/LazyParity.hs view
@@ -2,11 +2,6 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The HARD GATE for the lazy engine: for a bounded source the lazy result-must be BYTE-IDENTICAL to the eager result of the same ops (bounded joins and-aggregates route through the eager fast paths). Covers a LEFT-join pipeline-(join -> groupBy -> aggregate -> sort) and a pure groupBy pipeline.--} module LazyParity (tests) where import qualified Data.Map.Strict as M@@ -48,12 +43,10 @@ , ("plan", schemaType @Text) ] -{- | @n@ orders; ~10% reference a customer id with no matching customer row so-the LEFT join produces a Nothing group (exercises unmatched-row semantics).--}+-- | Orders frame where about 10% reference a customer with no matching row. ordersFrame :: Int -> D.DataFrame ordersFrame n =- D.fromNamedColumns+ D.fromColumns [ ("order_id", DI.fromList [0 .. n - 1]) , ( "customer_id"@@ -69,7 +62,7 @@ customersFrame :: Int -> D.DataFrame customersFrame m =- D.fromNamedColumns+ D.fromColumns [ ("customer_id", DI.fromList [0 .. m - 1]) , ("region", DI.fromList [T.pack ("r" ++ show (i `mod` 4)) | i <- [0 .. m - 1]]) , ("plan", DI.fromList [T.pack ("p" ++ show (i `mod` 3)) | i <- [0 .. m - 1]])@@ -84,9 +77,7 @@ removeFile csvPath return r -{- | LEFT join 20k orders to 257 customers -> groupBy region,plan--> sum(amount-discount), count -> sort revenue desc.--}+-- | Left-join orders to customers, then group, aggregate and sort. joinPipelineParity :: Test joinPipelineParity = TestCase $@@ -113,7 +104,7 @@ |> L.sortBy [("revenue", Descending)] |> L.runDataFrame assertEqual- "join pipeline lazy == eager (byte-identical)"+ "join pipeline lazy == eager" (show eager) (show lazy) @@ -138,7 +129,7 @@ |> L.sortBy [("total", Descending)] |> L.runDataFrame assertEqual- "groupBy pipeline lazy == eager (byte-identical)"+ "groupBy pipeline lazy == eager" (show eager) (show lazy) @@ -172,7 +163,7 @@ (show actual) where input =- D.fromNamedColumns+ D.fromColumns [ ("order_id", DI.fromList [1 .. 12 :: Int]) , ("customer_id", DI.fromList [0, 1, 2, 0, 1, 2, 0, 1, 2, 0, 1, 2 :: Int]) ,
tests/LazyParquet.hs view
@@ -87,7 +87,7 @@ assertEqual "filterAndProject" (2, 2) (D.dimensions actual) ) --- Test 5: multi-file glob — alltypes_plain*.parquet matches plain + snappy = 10 rows.+-- Test 5: multi-file glob matches plain + snappy = 10 rows. multiFileGlob :: Test multiFileGlob = TestCase
tests/LazyProjection.hs view
@@ -2,9 +2,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | A lazy scan reads only the columns its 'Schema' names, and rejects-self-contradictory 'ReadOptions' before opening the file.--}+-- | A lazy scan reads only the columns its schema names. module LazyProjection (tests) where import qualified Data.Map.Strict as M@@ -42,9 +40,7 @@ twoCols = Schema $ M.fromList [("id", schemaType @Int), ("dob", schemaType @Int)] -{- | The scan keeps only the schema's columns — and 'dob' is not a prefix of-the header, so a positional scheme could not express this.--}+-- | The scan keeps only the schema's columns, whatever their header position. scanProjects :: Test scanProjects = TestLabel "scan_projects" $ TestCase $ withCsv ',' $ \csvPath -> do df <- L.runDataFrame (L.scanCsv twoCols (T.pack csvPath))@@ -69,14 +65,11 @@ L.runDataFrame (L.scanCsvWith Fast.fastReadCsvWithOpts twoCols (T.pack csvPath)) assertEqual "same frame" slow fast -{- | 'scanSeparated' used to drop its separator on the floor: the reader was-built from the schema alone and always split on commas.--}+-- | scanSeparated must honour its separator, not always split on commas. scanHonoursSeparator :: Test scanHonoursSeparator = TestLabel "scan_honours_separator" $ TestCase $ withCsv ';' $ \csvPath -> do- -- With the separator ignored, the whole line is one field and the- -- schema's columns would not be found at all.+ -- With the separator ignored, the schema's columns are not found. df <- L.runDataFrame (L.scanSeparated ';' twoCols (T.pack csvPath)) assertEqual "dims" (2, 2) (D.dimensions df) assertEqual "columns" ["dob", "id"] (D.columnNames df)@@ -141,10 +134,7 @@ "cannot delimit fields" ] -{- | The SIMD reader's low-level entry takes its separator as a scanner byte,-so a 'columnSeparator' saying something else would be silently ignored.-Reject it instead.--}+-- | A column separator that disagrees with the scanner byte is rejected. rejectSeparatorMismatch :: Test rejectSeparatorMismatch = TestLabel "reject_separator_mismatch" $ TestCase $ do let opts = Csv.defaultReadOptions{Csv.columnSeparator = ';'}
tests/Learn/Denotation.hs view
@@ -1,12 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The "predict is the model's denotation" claim, discharged as a test rather-than left as a slogan: the @Expr@ that 'predict' compiles must evaluate to the-same numbers as the fitted record's own parameters, computed independently here-in plain Haskell. If @affineExpr@ / @argMinExpr@ ever drift from the record they-are built from, these fail.--}+-- | The Expr from predict must match the fitted parameters numerically. module Learn.Denotation (tests) where import qualified Data.Text as T@@ -22,23 +17,22 @@ import DataFrame.KMeans import DataFrame.LinearModel-import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double]-interpD df e = case interpret @Double df e of+interpD frame e = case interpret @Double frame e of Right (TColumn c) -> either (const []) VU.toList (toVector @Double @VU.Vector c) Left err -> error (show err) interpI :: D.DataFrame -> Expr Int -> [Int]-interpI df e = case interpret @Int df e of+interpI frame e = case interpret @Int frame e of Right (TColumn c) -> either (const []) VU.toList (toVector @Int @VU.Vector c) Left err -> error (show err) df :: D.DataFrame df =- D.fromNamedColumns+ D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [3 + 2 * a - 0.5 * b | (a, b) <- zip xs1 xs2])@@ -50,7 +44,7 @@ col :: D.DataFrame -> T.Text -> [Double] col d n = interpD d (F.col @Double n) --- | @interpret (predict m)@ must equal @intercept + Σ coefⱼ·featureⱼ@ from the record.+-- | @interpret (predict m)@ must equal @intercept + Σ coefⱼ·featureⱼ@. linearDenotation :: Test linearDenotation = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") df@@ -66,7 +60,7 @@ "linear: predict Expr matches the record's affine prediction" (and (zipWith (\a b -> abs (a - b) < 1e-9) native symbolic)) --- | @interpret (predict km)@ must equal the nearest-centroid label from @kmCenters@.+-- | @interpret (predict km)@ must equal the nearest-centroid label. kmeansDenotation :: Test kmeansDenotation = TestCase $ do let feats = ["x1", "x2"]@@ -88,7 +82,7 @@ transposeL [] = [] transposeL xss | any null xss = []- | otherwise = map head xss : transposeL (map tail xss)+ | otherwise = [x | (x : _) <- xss] : transposeL [xs | (_ : xs) <- xss] tests :: [Test] tests = [linearDenotation, kmeansDenotation]
tests/Learn/Ensembles.hs view
@@ -19,7 +19,6 @@ import Data.Maybe (isJust, isNothing, listToMaybe) import qualified Data.Vector.Unboxed as VU-import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double]@@ -34,14 +33,14 @@ clsDF :: D.DataFrame clsDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([-3, -2, -1, -0.5, 0.5, 1, 2, 3] :: [Double])) , ("label", DI.fromList ([0, 0, 0, 0, 1, 1, 1, 1] :: [Int])) ] blobs :: D.DataFrame blobs =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList ([0, 0.2, -0.1, 0.1, 8, 8.1, 7.9, 8.2] :: [Double])) , ("b", DI.fromList ([0, -0.1, 0.2, 0.0, 5, 5.2, 4.9, 5.1] :: [Double])) ]@@ -49,7 +48,7 @@ testGBMRegression :: Test testGBMRegression = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1 .. 12] :: [Double])) , ( "y"@@ -73,7 +72,7 @@ testGBMStaged :: Test testGBMStaged = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1 .. 8] :: [Double])) , ("y", DI.fromList ([1 .. 8] :: [Double])) ]@@ -110,7 +109,7 @@ testDBSCAN :: Test testDBSCAN = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList ([0, 0.1, 0.2, 5, 5.1, 5.2, 50] :: [Double])) , ("b", DI.fromList ([0, 0.1, 0.0, 5, 5.0, 5.1, 50] :: [Double])) ]@@ -132,7 +131,7 @@ testGridSearch :: Test testGridSearch = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1 .. 40] :: [Double])) , ( "y"@@ -152,18 +151,10 @@ assertBool "best score high" (gsBestScore res > 0.99) assertEqual "all configs scored" 3 (length (gsAll res)) -{- | Logistic boosting must recover a known conditional probability, not merely-rank it. A leaf set to the mean gradient instead of @Σg/Σh@ understeps every-step by at least 4x, leaving probabilities shrunk toward the base rate while the-ranking — and so any accuracy or AUC check — still looks healthy.--Each @x@ carries a fixed 20 rows of which exactly @round (20 * trueP x)@ are-positive, so the empirical conditional probability at every @x@ is 'trueP' and-the target is separable in rank but not in value.--}+-- | Logistic boosting must recover a known conditional probability. sigmoidCurveDF :: D.DataFrame sigmoidCurveDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList (concatMap (replicate group . fst) cells)) , ("label", DI.fromList (concatMap snd cells)) ]@@ -199,9 +190,81 @@ ("logistic boosting recovers the conditional probability " ++ show err) (err < 0.03) +gbmData :: D.DataFrame+gbmData =+ D.fromColumns+ [ ("x", DI.fromList ([fromIntegral i / 4 | i <- [1 .. 40 :: Int]] :: [Double]))+ ,+ ( "z"+ , DI.fromList+ ([fromIntegral ((i * 7) `mod` 11) | i <- [1 .. 40 :: Int]] :: [Double])+ )+ ,+ ( "y"+ , DI.fromList+ ( [ sin (fromIntegral i / 4) + fromIntegral ((i * 7) `mod` 11) * 0.1+ | i <- [1 .. 40 :: Int]+ ] ::+ [Double]+ )+ )+ ]++-- A constant base equal to the default initial score boosts the same trees.+testGBMBaseScore :: Test+testGBMBaseScore = TestCase $ do+ let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2}+ ys = interpD gbmData (F.col @Double "y")+ mean = sum ys / fromIntegral (length ys)+ plain = fit cfg (F.col @Double "y") gbmData+ based = fit cfg{gbBaseScore = Just (F.lit mean)} (F.col @Double "y") gbmData+ gap =+ maximum+ ( zipWith+ (\a b -> abs (a - b))+ (interpD gbmData (predict plain))+ (interpD gbmData (predict based))+ )+ assertBool+ ("base-score model matches the default, gap " ++ show gap)+ (gap < 1e-9)+ -- A per-row base is part of the prediction: shifting it shifts every score.+ let shifted = fit cfg{gbBaseScore = Just (F.col @Double "z")} (F.col @Double "y") gbmData+ noTrees =+ fit+ cfg{gbNEstimators = 0, gbBaseScore = Just (F.col @Double "z")}+ (F.col @Double "y")+ gbmData+ assertEqual+ "zero trees predict the base"+ (interpD gbmData (F.col @Double "z"))+ (interpD gbmData (predict noTrees))+ assertBool+ "base model still fits"+ (length (interpD gbmData (predict shifted)) == 40)++-- Sampling at 1 is inert; below 1 it changes the fit, fixed by the seed.+testGBMSubsample :: Test+testGBMSubsample = TestCase $ do+ let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2}+ scores :: GBConfig -> [Double]+ scores c = interpD gbmData (predict (fit c (F.col @Double "y") gbmData))+ assertEqual+ "subsample 1 / colsample 1 reproduce the default"+ (scores cfg)+ (scores cfg{gbSubsample = 1, gbColsample = 1})+ let half = cfg{gbSubsample = 0.5, gbColsample = 0.5, gbSeed = 3}+ assertEqual "deterministic in the seed" (scores half) (scores half)+ assertBool "sampling changes the fit" (scores half /= scores cfg)+ assertBool+ "a different seed changes the fit"+ (scores half /= scores half{gbSeed = 4})+ tests :: [Test] tests = [ testGBMRegression+ , testGBMBaseScore+ , testGBMSubsample , testGBMStaged , testGBMCalibration , testAdaBoost
tests/Learn/Metamorphic.hs view
@@ -1,15 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Metamorphic, invariance, and abstraction-law tests for the ML library.--These check how the fitted model and the metrics behave under transformations of-the *input* — duplicate / permute / scale / rename / add-constant-feature — where-the math dictates the answer. Each test computes the model on two genuinely-different (transformed) frames and asserts they agree, or pins a metric/scaler to-its defining law. None compares a value to itself; each can fail if the library-is wrong.--}+-- | Metamorphic and invariance tests: models must obey transformation laws. module Learn.Metamorphic (tests) where import qualified Data.Text as T@@ -33,8 +25,6 @@ import Test.HUnit --- Helpers -------------------------------------------------------------------- interpD :: D.DataFrame -> Expr Double -> [Double] interpD df e = case interpret @Double df e of Right (TColumn c) -> either (const []) VU.toList (toVector @Double @VU.Vector c)@@ -49,7 +39,7 @@ -- Build a regression frame from explicit feature/target lists. mkRegDF :: [(T.Text, [Double])] -> [Double] -> D.DataFrame mkRegDF feats ys =- D.fromNamedColumns+ D.fromColumns ([(n, DI.fromList vs) | (n, vs) <- feats] ++ [("y", DI.fromList ys)]) -- Base design: a noiseless linear target so OLS recovers a unique solution.@@ -64,12 +54,7 @@ fitBase :: D.DataFrame -> LinearRegressor fitBase = fit defaultLinearConfig (F.col @Double "y") --- Metamorphic: duplicate rows ------------------------------------------------{- | OLS minimises Σ(y-ŷ)²; duplicating every row scales the loss by 2 but leaves-the argmin (the coefficients/intercept) unchanged. Catches a solver that-weights rows by frame size or mishandles the normal equations.--}+-- | Duplicating every row leaves the OLS coefficients unchanged. testDuplicateRows :: Test testDuplicateRows = TestCase $ do let dupDF =@@ -85,12 +70,7 @@ "duplicate rows: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Metamorphic: permute rows ---------------------------------------------------{- | OLS is invariant to row order. We reverse the rows (a genuine non-identity-permutation) and require identical coefficients. Catches any order-dependence-in matrix assembly or solving.--}+-- | OLS is invariant to row order. testPermuteRows :: Test testPermuteRows = TestCase $ do let permDF =@@ -106,13 +86,7 @@ "permute rows: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Metamorphic: scale a feature ------------------------------------------------{- | Scaling feature x1 by k must scale that coefficient by 1/k while leaving-predictions (and every other coefficient + the intercept) invariant. Both-sides are genuine fits on different frames. Catches a regressor that drops or-mis-applies a feature's scale.--}+-- | Scaling a feature by k scales its coefficient by 1/k and keeps predictions. testScaleFeature :: Test testScaleFeature = TestCase $ do let k = 10.0@@ -123,7 +97,7 @@ coef1 = VU.toList (regCoef m1) assertBool "scale feature: x1 coefficient scaled by 1/k"- (close 1e-7 (head coef0 / k) (head coef1))+ (close 1e-7 (VU.head (regCoef m0) / k) (VU.head (regCoef m1))) assertBool "scale feature: x2 coefficient unchanged" (close 1e-7 (coef0 !! 1) (coef1 !! 1))@@ -135,12 +109,7 @@ p1 = interpD scaledDF (predict m1) assertBool "scale feature: predictions invariant" (closeList 1e-6 p0 p1) --- Metamorphic: rename columns -------------------------------------------------{- | Renaming feature columns is a pure relabelling: coefficients, intercept, and-predictions are identical and the fitted feature names track the rename.-Catches any hidden dependence on specific column-name strings.--}+-- | Renaming feature columns only relabels coefficients and feature names. testRenameColumns :: Test testRenameColumns = TestCase $ do let renamedDF = mkRegDF [("feat_a", baseX1), ("feat_b", baseX2)] baseY@@ -161,14 +130,7 @@ p1 = interpD renamedDF (predict m1) assertBool "rename: predictions unchanged" (closeList 1e-9 p0 p1) --- Metamorphic: irrelevant constant feature ------------------------------------{- | A constant feature carries no variance, so OLS predictions for the real-features must be unchanged when one is added. (Its own weight is absorbed into-the intercept and is not separately identifiable, so we pin predictions, not-the weight.) Catches a fit whose answer drifts when a degenerate column is-present.--}+-- | A constant feature must not change predictions for the real features. testConstantFeatureIgnored :: Test testConstantFeatureIgnored = TestCase $ do let withConst =@@ -189,20 +151,12 @@ let coef1 = VU.toList (regCoef m1) assertBool "constant feature: x1 coefficient unchanged"- (close 1e-6 (head (VU.toList (regCoef m0))) (head coef1))+ (close 1e-6 (VU.head (regCoef m0)) (VU.head (regCoef m1))) assertBool "constant feature: x2 coefficient unchanged" (close 1e-6 (VU.toList (regCoef m0) !! 1) (coef1 !! 1)) --- Metamorphic: split then concatenate ----------------------------------------{- | Genuinely split the frame into two row-range sub-frames and concatenate them-back with the DataFrame @<>@ (row concatenation). The rebuilt frame has the-same rows, so the fit must be identical — but this actually exercises the-concatenation code path, so it catches a @<>@ that drops, duplicates, or-mis-aligns rows/columns (a plain @take h ++ drop h@ would be the identity and-test nothing).--}+-- | Splitting a frame and rejoining it with <> must give an identical fit. testSplitConcat :: Test testSplitConcat = TestCase $ do let n = length baseX1@@ -226,14 +180,7 @@ "split/concat: intercept unchanged" (close 1e-9 (regIntercept m0) (regIntercept m1)) --- Training/inference parity: column order ------------------------------------{- | The order of feature columns in the frame must not change predictions.-We fit on a frame whose columns are declared in the opposite order and require-the prediction expression to produce the same numbers on the original frame.-(predict compiles to a name-keyed affine Expr, so it should be order-free.)-Catches a positional, rather than name-keyed, feature mapping.--}+-- | The order of feature columns must not change predictions. testColumnOrderParity :: Test testColumnOrderParity = TestCase $ do let swappedDF = mkRegDF [("x2", baseX2), ("x1", baseX1)] baseY@@ -300,7 +247,7 @@ a0 = accuracy preds truth a1 = accuracy (VU.reverse preds) (VU.reverse truth) assertBool "accuracy: permutation invariant" (close 1e-12 a0 a1)- -- Sanity: the metric is non-trivial here (not 0 or 1), so invariance is meaningful.+ -- Sanity: the metric is non-trivial here, so invariance is meaningful. assertBool "accuracy: non-degenerate baseline" (a0 > 0 && a0 < 1) testR2Anchors :: Test@@ -314,9 +261,7 @@ let worse = VU.map (+ 100) truth assertBool "r2: far-off prediction is negative" (r2 worse truth < 0) -{- | r² on a fitted noiseless linear model equals 1, and rmse equals 0. Ties the-metric law to the model: if the fit is exact, the metric must say so.--}+-- | A noiseless linear fit has r2 equal to 1 and rmse equal to 0. testR2OfFittedModel :: Test testR2OfFittedModel = TestCase $ do let m = fitBase baseDF@@ -329,7 +274,7 @@ testTargetNeverAFeature = TestCase $ do let n = 40 :: Int leakDF =- D.fromNamedColumns+ D.fromColumns [ ("y", DI.fromList [if even i then 1.0 else 0.0 :: Double | i <- [0 .. n - 1]]) , ("a", DI.fromList [fromIntegral (i `div` 4) :: Double | i <- [0 .. n - 1]]) , ("b", DI.fromList [fromIntegral (i `mod` 3) :: Double | i <- [0 .. n - 1]])
tests/Learn/MetricsTests.hs view
@@ -17,8 +17,10 @@ import DataFrame.PCA import DataFrame.Transform +import Data.List (sortBy)+import Data.Ord (comparing) import qualified Data.Vector.Unboxed as VU-import DataFrame.Model (fit, predict)+import System.Random (mkStdGen, randomRs) import Test.HUnit close :: Double -> Double -> Double -> Bool@@ -30,7 +32,7 @@ reg :: D.DataFrame reg =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1 .. 20] :: [Double])) , ( "y"@@ -117,9 +119,63 @@ assertBool "pipeline produced a frame" (D.columnNames out /= []) assertBool "pipeline has pc1 column" ("pc1" `elem` D.columnNames out) +-- | Mann-Whitney AUC computed from ranks over lists, the reference for rocAuc.+referenceAuc :: [Double] -> [Double] -> Double+referenceAuc scores truth+ | nPos == 0 || nNeg == 0 = 0.5+ | otherwise = (rankSum - nPos * (nPos + 1) / 2) / (nPos * nNeg)+ where+ sorted = sortBy (comparing snd) (zip [0 :: Int ..] scores)+ ranks = map snd (sortBy (comparing fst) (assign 1 sorted))+ assign _ [] = []+ assign start grp@((_, v) : _) =+ let (tied, rest) = span ((== v) . snd) grp+ k = length tied+ avg = fromIntegral (sum [start .. start + k - 1]) / fromIntegral k+ in [(i, avg) | (i, _) <- tied] ++ assign (start + k) rest+ rankSum = sum [r | (y, r) <- zip truth ranks, y == 1]+ nPos = fromIntegral (length (filter (== 1) truth))+ nNeg = fromIntegral (length truth) - nPos++testRocAucMatchesReference :: Test+testRocAucMatchesReference = TestCase $ do+ let grid =+ [ (n, levels, seed)+ | n <- [1, 2, 17, 1000]+ , levels <- [2, 7, 100000]+ , seed <- [1, 2, 3]+ ]+ sequence_+ [ assertEqual+ ("n=" ++ show n ++ " levels=" ++ show levels ++ " seed=" ++ show seed)+ (referenceAuc scores truth)+ (rocAuc (VU.fromList scores) (VU.fromList truth))+ | (n, levels, seed) <- grid+ , let scores = map fromIntegral (take n (randomRs (0, levels - 1 :: Int) (mkStdGen seed)))+ truth = map fromIntegral (take n (randomRs (0, 1 :: Int) (mkStdGen (seed + 100))))+ ]++testStratifiedFolds :: Test+testStratifiedFolds = TestCase $ do+ let labels =+ VU.fromList (map fromIntegral (take 1003 (randomRs (0, 1 :: Int) (mkStdGen 7))))+ folds = stratifiedFoldIds 5 42 labels+ perFold c =+ [ VU.length (VU.filter id (VU.zipWith (\f l -> f == k && l == c) folds labels))+ | k <- [0 .. 4]+ ]+ spread xs = maximum xs - minimum xs+ assertEqual "one fold per row" (VU.length labels) (VU.length folds)+ assertBool "folds in range" (VU.all (\f -> f >= 0 && f < 5) folds)+ assertBool "positives balanced" (spread (perFold 1) <= 1)+ assertBool "negatives balanced" (spread (perFold 0) <= 1)+ assertEqual "deterministic" folds (stratifiedFoldIds 5 42 labels)+ tests :: [Test] tests =- [ testRegressionMetrics+ [ testRocAucMatchesReference+ , testStratifiedFolds+ , testRegressionMetrics , testMulticlassMetrics , testRocAuc , testReports
tests/Learn/Models.hs view
@@ -21,11 +21,9 @@ import Control.Exception (evaluate, try) import Data.List (isInfixOf) import qualified Data.Map.Strict as M-import qualified Data.Text as T import qualified Data.Vector as V import qualified Data.Vector.Unboxed as VU import DataFrame.Errors (DataFrameException)-import DataFrame.Model (fit, predict) import Test.HUnit interpD :: D.DataFrame -> Expr Double -> [Double]@@ -43,7 +41,7 @@ regDF :: D.DataFrame regDF =- D.fromNamedColumns+ D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2])@@ -57,7 +55,7 @@ flowerDF :: D.DataFrame flowerDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([-5, -4, -3, 0, 0.3, 0.6, 5, 6, 7] :: [Double])) , ( "flower"@@ -67,7 +65,7 @@ clsDF :: D.DataFrame clsDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([-3, -2, -1, -0.5, 0.5, 1, 2, 3] :: [Double])) , ("label", DI.fromList ([0, 0, 0, 0, 1, 1, 1, 1] :: [Int])) ]@@ -83,19 +81,15 @@ truth = interpD regDF (F.col @Double "y") assertBool "OLS expr matches y" (and (zipWith (close 1e-6) preds truth)) -{- | A regression frame whose feature column @x@ is stored as @Int@; the target-@y@ is @Double@. Fitting must reject the Int feature at the boundary.--}+-- | Regression frame whose feature x is stored as Int. intFeatureDF :: D.DataFrame intFeatureDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList @Int [1 .. 8]) , ("y", DI.fromList @Double [2, 4, 6, 8, 10, 12, 14, 16]) ] -{- | The fit boundary mandates Double: a non-Double feature column is a fit-time-error (naming the column and pointing at the fix), not a silent coercion.--}+-- | A non-Double feature column is a fit-time error, not a silent coercion. testFitRequiresDouble :: Test testFitRequiresDouble = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") intFeatureDF@@ -108,19 +102,15 @@ assertBool "error names the offending column x" ("x" `isInfixOf` msg) Right _ -> assertFailure "expected a Double-input error for the Int feature column" -{- | A regression frame whose feature @x@ is nullable (@Maybe Double@). Fitting-must reject it rather than letting nulls poison the solve into NaN.--}+-- | Regression frame whose feature x is nullable. nullableFeatureDF :: D.DataFrame nullableFeatureDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList @(Maybe Double) (Nothing : map Just [2, 3, 4, 5, 6, 7, 8])) , ("y", DI.fromList @Double [2, 4, 6, 8, 10, 12, 14, 16]) ] -{- | The fit boundary rejects a nullable (Maybe Double) input column with an-actionable error, instead of silently producing NaN coefficients.--}+-- | Fitting rejects a nullable input column with an actionable error. testFitRejectsNullable :: Test testFitRejectsNullable = TestCase $ do let m = fit defaultLinearConfig (F.col @Double "y") nullableFeatureDF@@ -153,9 +143,7 @@ let probs = logisticProbExprs m assertBool "prob exprs present for both classes" (M.size probs == 2) -{- | Logistic regression must work with any enum label type, not just Int —-multiclass classification into a user-defined sum type.--}+-- | Logistic regression must work with any enum label type, not just Int. testLogisticEnumLabel :: Test testLogisticEnumLabel = TestCase $ do let m = fit defaultLogisticConfig (F.col @Flower "flower") flowerDF@@ -206,14 +194,14 @@ testKMeans :: Test testKMeans = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList ([0, 0.1, 0.2, 10, 10.1, 10.2] :: [Double])) , ("b", DI.fromList ([0, 0.1, 0, 10, 10, 10.1] :: [Double])) ] cfg = defaultKMeansConfig{kmK = 2, kmNInit = 5, kmSeed = 1} m = fit cfg [F.col @Double "a", F.col @Double "b"] df labels = VU.toList (kmLabels m)- assertBool "two blobs split" (head labels /= last labels)+ assertBool "two blobs split" (VU.head (kmLabels m) /= VU.last (kmLabels m)) assertEqual "two centers" 2 (V.length (kmCenters m)) let m2 = fit cfg [F.col @Double "a", F.col @Double "b"] df assertEqual "kmeans deterministic" (kmCenters m) (kmCenters m2)
tests/Learn/Segmented.hs view
@@ -44,7 +44,7 @@ -- | A two-segment frame: y = 2x+1 on group "a", y = -3x+5 on group "b". twoSegDF :: D.DataFrame twoSegDF =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("x", DI.fromList xs) ,@@ -57,7 +57,9 @@ segByKey :: [T.Text] -> SegmentedModel a LinearRegressor -> Segment LinearRegressor-segByKey k m = head [s | s <- smSegments m, segKey s == k]+segByKey k m = case [s | s <- smSegments m, segKey s == k] of+ (s : _) -> s+ [] -> error ("segByKey: no segment for " ++ show k) lowFloor :: Segmented cfg -> Segmented cfg lowFloor s = s{segMinRows = 3}@@ -86,7 +88,7 @@ let m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") twoSegDF fb = smFallback m unseen =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (["c"] :: [T.Text])) , ("x", DI.fromList ([2.0] :: [Double])) , ("y", DI.fromList ([0.0] :: [Double]))@@ -103,7 +105,7 @@ [ "logistic base composes" ~: do let labels = replicate 4 ("pos" :: T.Text) ++ replicate 4 "neg" df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double])) , ("label", DI.fromList labels)@@ -122,7 +124,7 @@ selectionTests = [ "segmentOn picks only the named column" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("h", DI.fromList (cycle3 8)) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double]))@@ -136,7 +138,7 @@ assertEqual "only g" ["g"] (smCatCols m) , "cardinality cap skips high-cardinality text" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("k", DI.fromList (map (T.pack . show) [1 .. 8 :: Int])) , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double]))@@ -156,7 +158,7 @@ [Just "a", Just "a", Just "a", Nothing, Just "b", Just "b", Just "b", Nothing] :: [Maybe T.Text] df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList gcol) , ("x", DI.fromList ([1, 2, 3, 9, 1, 2, 3, 9] :: [Double])) , ("y", DI.fromList ([3, 5, 7, 0, 2, -1, -4, 0] :: [Double]))@@ -180,7 +182,7 @@ guardTests = [ "int feature error points at toDouble" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("n", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Int])) , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double]))@@ -194,7 +196,7 @@ Right _ -> assertFailure "expected an actionable error for the Int feature" , "nullable feature error points at dropping" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ( "z"@@ -214,7 +216,7 @@ Right _ -> assertFailure "expected an actionable error for the nullable feature" , "well-typed frame fits after casting/dropping" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ replicate 4 "b")) , ("n", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double])) , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double]))@@ -227,7 +229,7 @@ diagnosticTests = [ "undersized segment falls back, diagnostics correct" ~: do let df =- D.fromNamedColumns+ D.fromColumns [ ("g", DI.fromList (replicate 4 ("a" :: T.Text) ++ ["b"])) , ("x", DI.fromList ([1, 2, 3, 4, 9] :: [Double])) , ("y", DI.fromList ([3, 5, 7, 9, 0] :: [Double]))@@ -273,15 +275,14 @@ -- lambda = 0: independent OLS (matches recovery test). assertBool "lambda0 a slope 2" (close 1e-9 (slope ["a"] m0) 2) assertBool "lambda0 b slope -3" (close 1e-9 (slope ["b"] m0) (-3))- -- lambda -> inf: both segments converge to the n_g-weighted reference- -- (raw slope -0.5, raw intercept 3.0 for this balanced fixture).+ -- lambda -> inf: both segments converge to the weighted reference. assertBool "lambdaInf a slope ~ -0.5" (close 1e-3 (slope ["a"] mInf) (-0.5)) assertBool "lambdaInf b slope ~ -0.5" (close 1e-3 (slope ["b"] mInf) (-0.5)) assertBool "lambdaInf a intercept ~ 3" (close 1e-3 (inter ["a"] mInf) 3.0) assertBool "lambdaInf segments coincide" (close 1e-3 (slope ["a"] mInf) (slope ["b"] mInf))- -- interior: a's slope sits strictly between its OLS (2) and the ref (-0.5).+ -- interior: a's slope sits strictly between its OLS and the ref. assertBool "interior between OLS and ref" (slope ["a"] mMid < 2 && slope ["a"] mMid > (-0.5))
tests/Learn/SklearnParity.hs view
@@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Parity tests against scikit-learn on clean Kaggle-style datasets; reference-values in @data/ml/golden.json@ come from @scripts/gen_sklearn_golden.py@.-Closed-form models get tight coefficient parity; iterative ones an accuracy floor.--}+-- | Parity tests against scikit-learn using reference values in golden.json. module Learn.SklearnParity (tests) where import Data.Aeson (Value (..), decodeFileStrict')@@ -14,7 +11,6 @@ import qualified Data.Text as T import qualified Data.Vector as V import qualified Data.Vector.Unboxed as VU-import DataFrame.Model (fit, predict) import Test.HUnit import qualified DataFrame as D
tests/Learn/Synthesis.hs view
@@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Feature-synthesis enumerator: it should recover small exact features-(@x²@, @a/b@) from the example rows, score them near-perfectly, and be-deterministic.--}+-- | The synthesis enumerator recovers small exact features deterministically. module Learn.Synthesis (tests) where import Assertions (assertExpectException)@@ -16,7 +13,7 @@ quad :: D.DataFrame quad =- D.fromNamedColumns+ D.fromColumns [ ("x", D.fromList xs) , ("y", D.fromList (map (\x -> x * x) xs)) ]@@ -25,7 +22,7 @@ ratio :: D.DataFrame ratio =- D.fromNamedColumns+ D.fromColumns [ ("a", D.fromList ([2, 6, 12, 20, 30, 42] :: [Double])) , ("b", D.fromList ([1, 2, 3, 4, 5, 6] :: [Double])) , ("y", D.fromList ([2, 3, 4, 5, 6, 7] :: [Double]))@@ -74,11 +71,7 @@ (D.prettyPrint (sfExpr a)) (D.prettyPrint (sfExpr b)) -{- | A wide frame at the default 'synMaxSize' refuses instead of exhausting the-heap. 'synBankCap' caps what is kept, not what is generated, so the layers past-size 4 used to allocate tens of gigabytes and kill the process — which no test-can catch, because there is no process left to fail.--}+-- | A wide frame at the default synMaxSize refuses, not exhausts the heap. refusesOversizedSearch :: Test refusesOversizedSearch = TestCase@@ -100,7 +93,7 @@ -- | 12 features over 3000 rows: the shape that killed the kernel. wide :: D.DataFrame wide =- D.fromNamedColumns+ D.fromColumns ( ("y", D.fromList (map (\i -> fromIntegral (i `mod` 7) :: Double) idx)) : [ ( "f" <> T.pack (show c) , D.fromList (map (\i -> fromIntegral ((i * c) `mod` 13) :: Double) idx)
tests/Learn/TypedModel.hs view
@@ -3,11 +3,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | 'fit'/'predict' are frame-aware. Training on a 'TypedDataFrame' with a typed-target ('DT.col') returns a 'Fitted' carrying the schema, and 'predict' on it-yields a typed 'TExpr'; training on an untyped 'DataFrame' with an untyped target-('F.col') returns the bare model and an 'Expr'. Both paths must agree numerically.--}+-- | Typed and untyped fit/predict paths must agree numerically. module Learn.TypedModel (tests) where import Data.Maybe (fromJust)@@ -40,7 +36,7 @@ regDF :: D.DataFrame regDF =- D.fromNamedColumns+ D.fromColumns [ ("x1", DI.fromList xs1) , ("x2", DI.fromList xs2) , ("y", DI.fromList [2 * a - 3 * b + 1 | (a, b) <- zip xs1 xs2])@@ -65,9 +61,7 @@ and (zipWith (close 1e-12) (VU.toList (regCoef a)) (VU.toList (regCoef b))) && close 1e-12 (regIntercept a) (regIntercept b) -{- | 'fit' on a 'TypedDataFrame' returns a 'Fitted' whose model matches the-bare model from 'fit' on the equivalent 'DataFrame'.--}+-- | Typed fit returns a model matching the fit on the equivalent DataFrame. testTypedFitParity :: Test testTypedFitParity = TestCase $ do let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF@@ -76,14 +70,12 @@ "fittedModel of typed fit == untyped fit" (sameModel (fittedModel onTyped) onUntyped) -{- | 'predict' on the typed fit yields a 'TExpr Houses Double'; unwrapped and-interpreted it matches 'predict' on the untyped fit.--}+-- | Typed predict matches predict on the untyped fit once interpreted. testTypedPredict :: Test testTypedPredict = TestCase $ do let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF onUntyped = fit defaultLinearConfig (F.col @Double "y") regDF- typedExpr = DT.unTExpr (predict onTyped) -- :: Expr Double via TExpr Houses Double+ typedExpr = DT.unTExpr (predict onTyped) untypedExpr = predict onUntyped -- :: Expr Double truth = interpD regDF (F.col @Double "y") assertBool
tests/Main.hs view
@@ -34,13 +34,17 @@ import qualified Operations.Aggregations import qualified Operations.Apply import qualified Operations.Core+import qualified Operations.CountDistinct import qualified Operations.Derive+import qualified Operations.ExprParse import qualified Operations.Filter import qualified Operations.GroupBy import qualified Operations.Inference import qualified Operations.InsertColumn+import qualified Operations.InterpreterKernels import qualified Operations.Join import qualified Operations.Merge+import qualified Operations.MonomorphicLoops import qualified Operations.Nullable import qualified Operations.NullableHashing import qualified Operations.ParallelGroupBy@@ -48,12 +52,16 @@ import qualified Operations.Provenance import qualified Operations.ReadCsv import qualified Operations.Record+import qualified Operations.RowWise import qualified Operations.SetOps import qualified Operations.Shuffle import qualified Operations.Sort+import qualified Operations.SortMask import qualified Operations.Statistics import qualified Operations.Subset import qualified Operations.Take+import qualified Operations.TextKernels+import qualified Operations.TopBy import qualified Operations.Typing import qualified Operations.VectorKernel import qualified Operations.Window@@ -83,9 +91,14 @@ ++ Learn.SklearnParity.tests ++ Learn.Synthesis.tests ++ Learn.MetricsTests.tests+ ++ Operations.ExprParse.tests ++ Learn.Metamorphic.tests ++ Learn.Segmented.tests ++ Operations.Aggregations.tests+ ++ Operations.CountDistinct.tests+ ++ Operations.SortMask.tests+ ++ Operations.TextKernels.tests+ ++ Operations.TopBy.tests ++ Operations.Apply.tests ++ Operations.Core.tests ++ Operations.Derive.tests@@ -112,6 +125,9 @@ ++ Operations.Take.tests ++ Operations.Typing.tests ++ Operations.VectorKernel.tests+ ++ Operations.InterpreterKernels.tests+ ++ Operations.MonomorphicLoops.tests+ ++ Operations.RowWise.tests ++ Operations.Window.tests ++ Functions.tests ++ IO.CSV.tests
tests/Monad.hs view
@@ -30,7 +30,7 @@ let finalRowCount = D.nRows finalDf let realRate = roundToTwoPlaces $ fromIntegral finalRowCount / fromIntegral rowCount let diff = abs $ expectedRate - realRate- -- calculates the 99.99% confidence interval (quickcheck runs 100 tests, aim for 1/10000)+ -- 99.99% confidence interval (quickcheck runs 100 tests, 1/10000) let tolerance' = 3.89 * sqrt (expectedRate * (1 - expectedRate) / fromIntegral rowCount) assert (diff <= tolerance') @@ -41,7 +41,7 @@ verbFixture :: DataFrame verbFixture =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList ([3, 1, 2] :: [Int])) , ("B", DI.fromList (["x", "y", "z"] :: [T.Text])) , ("C", DI.fromList ([1.0, 2.0, 3.0] :: [Double]))
tests/Operations/Aggregations.hs view
@@ -25,14 +25,14 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values foldAggregation :: Test foldAggregation = TestCase ( assertEqual "Counting elements after grouping gives correct numbers"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6 :: Int, 3, 3]) ]@@ -49,7 +49,7 @@ TestCase ( assertEqual "countAll gives per-group row counts without a column argument"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("n", DI.fromList [6 :: Int, 3, 3]) ]@@ -66,7 +66,7 @@ TestCase ( assertEqual "Typed countAll gives per-group row counts"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("n", DI.fromList [6 :: Int, 3, 3]) ]@@ -93,7 +93,7 @@ TestCase ( assertEqual "Typed counting elements after grouping gives correct numbers"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2_count", DI.fromList [6 :: Int, 3, 3]) ]@@ -120,7 +120,7 @@ TestCase ( assertEqual "Mean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6.5 :: Double, 8.0, 5.0]) ]@@ -137,7 +137,7 @@ TestCase ( assertEqual "Typed ean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2_mean", DI.fromList [6.5 :: Double, 8.0, 5.0]) ]@@ -164,7 +164,7 @@ TestCase ( assertEqual "Mean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [6.5 :: Double, 8.0, 5.0]) ]@@ -183,7 +183,7 @@ TestCase ( assertEqual "Mean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [13 :: Double, 16, 10]) ]@@ -200,7 +200,7 @@ TestCase ( assertEqual "Mean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [12 :: Double, 9, 6]) ]@@ -220,7 +220,7 @@ TestCase ( assertEqual "Mean works for ints"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [1 :: Int, 2, 3]) , ("test2", DI.fromList [24 :: Int, 18, 12]) ]@@ -238,7 +238,7 @@ TestCase ( assertEqual "Aggregation on DataFrame with no rows"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList ([] :: [Int])) , ("sum(test2)", DI.fromList ([] :: [Int])) ]@@ -261,7 +261,7 @@ 3 ( D.nRows ( D.distinct- ( D.fromNamedColumns+ ( D.fromColumns [ ("x", DI.fromList [1 :: Int, 1, 2, 2, 3]) , ("y", DI.fromList [10 :: Int, 10, 20, 20, 30]) ]@@ -278,7 +278,7 @@ 3 ( D.nRows ( D.distinct- ( D.fromNamedColumns+ ( D.fromColumns [ ("x", DI.fromList [1 :: Int, 2, 3]) , ("y", DI.fromList [10 :: Int, 20, 30]) ]@@ -295,17 +295,17 @@ 1 ( D.nRows ( D.distinct- ( D.fromNamedColumns+ ( D.fromColumns [("x", DI.fromList [42 :: Int, 42, 42, 42])] ) ) ) ) --- groupBy on an Optional (nullable) column: Nothing values form their own group.+-- groupBy on a nullable column: Nothing values form their own group. optGroupByDf :: D.DataFrame optGroupByDf =- D.fromNamedColumns+ D.fromColumns [ ("key", DI.fromList [Just 1 :: Maybe Int, Just 1, Just 2, Nothing, Nothing]) , ("val", DI.fromList [10 :: Int, 20, 30, 40, 50]) ]
tests/Operations/Apply.hs view
@@ -34,7 +34,7 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values applyBoxedToUnboxed :: Test applyBoxedToUnboxed =@@ -93,14 +93,14 @@ ) where nullableApplyData =- D.fromNamedColumns [("test", DI.fromList ["", "b" :: String])]+ D.fromColumns [("test", DI.fromList ["", "b" :: String])] applyManyOnlyGivenFields :: Test applyManyOnlyGivenFields = TestCase ( assertEqual "Applies function to many fields"- ( D.fromNamedColumns+ ( D.fromColumns ( map (,D.fromList $ replicate 26 (1 :: Integer)) ["test4", "test6"] ++ -- All other fields should have their original values.@@ -119,7 +119,7 @@ TestCase ( assertEqual "Applies function to many fields"- ( D.fromNamedColumns+ ( D.fromColumns (map (,D.fromList $ replicate 26 (1 :: Integer)) ["test4", "test6", "test8"]) ) ( D.select ["test4", "test6", "test8"] $@@ -135,7 +135,7 @@ TestCase ( assertEqual "Unboxes fields when necessary"- ( D.fromNamedColumns+ ( D.fromColumns (map (,D.fromList $ replicate 26 (1 :: Int)) ["test4", "test6", "test8"]) ) ( D.select ["test4", "test6", "test8"] $@@ -229,7 +229,7 @@ imputeData :: D.DataFrame imputeData =- D.fromNamedColumns+ D.fromColumns [ ("opt", DI.fromList [Just (1 :: Int), Nothing, Just 3]) , ("plain", DI.fromList [10 :: Int, 20, 30]) ]
tests/Operations/Core.hs view
@@ -13,7 +13,7 @@ testData :: D.DataFrame testData =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList ([1 .. 3] :: [Int])) , ("B", DI.fromList ['a' .. 'c']) ]@@ -70,15 +70,13 @@ TestCase ( assertEqual "null cell preserves row alignment"- ( D.fromNamedColumns+ ( D.fromColumns [("A", DI.fromList ([Just 1, Nothing, Just 3] :: [Maybe Int]))] ) (D.fromRows ["A"] [[D.toAny (1 :: Int)], [Null], [D.toAny (3 :: Int)]]) ) -{- | An all-null column has as many rows as it was given. Collapsing it to an-empty column silently truncates the frame.--}+-- | An all-null column keeps the row count it was given. fromRowsAllNullColumnKeepsRows :: Test fromRowsAllNullColumnKeepsRows = TestCase@@ -88,14 +86,12 @@ (D.nRows (D.fromRows ["A"] [[Null], [Null], [Null]])) ) -{- | A frame with a null survives the round trip at full length. Guards the-alignment invariant through 'toRowList' as well as 'fromRows'.--}+-- | A frame with a null survives the round trip at full length. fromRowsRoundTripsWithNulls :: Test fromRowsRoundTripsWithNulls = TestCase ( let df =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList ([Just 1, Nothing, Just 3] :: [Maybe Int])) , ("B", DI.fromList (["x", "y", "z"] :: [T.Text])) ]
+ tests/Operations/CountDistinct.hs view
@@ -0,0 +1,213 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-}++module Operations.CountDistinct where++import qualified Data.ByteString as BS+import qualified Data.Map.Strict as M+import qualified Data.Set as S+import qualified Data.Text as T+import qualified Data.Text.Array as A+import qualified Data.Text.Encoding as TE+import qualified Data.Vector.Unboxed as VU+import qualified DataFrame as D+import qualified DataFrame.Functions as F+import qualified DataFrame.Internal.Column as DI++import Data.Bits (shiftR, xor)+import Data.Function ((&))+import Data.Int (Int32, Int64)+import Data.Text.Internal (Text (Text))+import DataFrame.Expression.Operators (as)+import DataFrame.Internal.Aggregation.Kernel.Distinct (+ countDistinctColumn,+ countDistinctGroups,+ )+import DataFrame.Internal.Data.PackedText (mkPackedContiguous)+import Test.HUnit++-- | Deterministic pseudo-random stream (splitmix-style).+stream :: Int -> Int -> [Int]+stream seed n = take n (drop 1 (map mix (iterate (+ (-7046029254386353131)) seed)))+ where+ mix z0 =+ let z1 = (z0 `xor` (z0 `shiftR` 30)) * (-4658895280553007687)+ z2 = (z1 `xor` (z1 `shiftR` 27)) * (-7723592293110705685)+ in z2 `xor` (z2 `shiftR` 31)++-- | Values in @[0, range)@.+draws :: Int -> Int -> Int -> [Int]+draws seed range n = map (`mod` range) (stream seed n)++refGlobal :: (Ord a) => [a] -> Int+refGlobal = S.size . S.fromList++refGrouped :: (Ord a) => [Int] -> [a] -> Int -> [Int]+refGrouped gs xs nGroups =+ let m = M.fromListWith S.union [(g, S.singleton x) | (g, x) <- zip gs xs]+ in [maybe 0 S.size (M.lookup g m) | g <- [0 .. nGroups - 1]]++-- | Packed UTF-8 column of the given texts.+packed :: [T.Text] -> DI.Column+packed ts =+ let bytes = BS.concat (map TE.encodeUtf8 ts)+ offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts)+ Text arr _ _ = TE.decodeUtf8 bytes+ in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs))++word :: Int -> T.Text+word k = "w" <> T.pack (show k)++globalCase :: String -> DI.Column -> Int -> Test+globalCase label col expected =+ TestCase (assertEqual label (Just expected) (countDistinctColumn col))++groupedCase :: String -> [Int] -> Int -> DI.Column -> [Int] -> Test+groupedCase label gs nGroups col expected =+ TestCase+ ( assertEqual+ label+ (Just (VU.fromList expected))+ (countDistinctGroups (VU.fromList gs) nGroups col)+ )++-- Sizes straddle the parallel threshold and force table growth.+sizes :: [Int]+sizes = [0, 1, 17, 5000, 300000]++intGlobal :: [Test]+intGlobal =+ [ globalCase+ ("Int n=" ++ show n ++ " r=" ++ show r)+ (DI.fromList xs)+ (refGlobal xs)+ | n <- sizes+ , r <- [3, n `div` 3 + 1, maxBound]+ , let xs = draws n r n+ ]++-- 'minBound' is the kernel's empty-slot marker, so it must still count once.+intSentinel :: Test+intSentinel =+ let xs = take 300000 (cycle [minBound, 0, -1, maxBound, minBound, 7 :: Int])+ in globalCase "Int with minBound" (DI.fromList xs) (refGlobal xs)++narrowInts :: [Test]+narrowInts =+ [ let xs = map fromIntegral (draws 1 100000 250000) :: [Int64]+ in globalCase "Int64" (DI.fromList xs) (refGlobal xs)+ , let xs = map fromIntegral (draws 2 100000 250000) :: [Int32]+ in globalCase "Int32" (DI.fromList xs) (refGlobal xs)+ ]++doubles :: Test+doubles =+ let xs =+ [-0.0, 0.0, 1.5, 1.5, -2.25, 1 / 0, -(1 / 0)]+ ++ map fromIntegral (draws 3 50 1000) ::+ [Double]+ in globalCase "Double, -0.0 joins 0.0" (DI.fromList xs) (refGlobal xs)++texts :: [Test]+texts =+ [ globalCase ("Text boxed n=" ++ show n) (DI.fromList ts) (refGlobal ts)+ | n <- sizes+ , let ts = map word (draws 4 (n `div` 4 + 1) n)+ ]+ ++ [ globalCase ("Text packed n=" ++ show n) (packed ts) (refGlobal ts)+ | n <- sizes+ , let ts = map word (draws 5 (n `div` 4 + 1) n) ++ ["", "", "é", "é"]+ ]++grouped :: [Test]+grouped =+ [ groupedCase+ ("Int grouped n=" ++ show n ++ " g=" ++ show ng)+ gs+ ng+ (DI.fromList xs)+ (refGrouped gs xs ng)+ | n <- sizes+ , ng <- [1, 7, 1000]+ , let gs = draws (n + 11) ng n+ , let xs = draws (n + 13) (n `div` 2 + 1) n+ ]+ ++ [ groupedCase "Text grouped" gs 50 (packed ts) (refGrouped gs ts 50)+ | let n = 300000+ , let gs = draws 21 50 n+ , let ts = map word (draws 22 20000 n)+ ]+ -- Enough groups to take the shared atomic-count path.+ ++ [ groupedCase+ "Int grouped, wide domain"+ gs+ ng+ (DI.fromList xs)+ (refGrouped gs xs ng)+ | let n = 1500000+ , let ng = 600000+ , let gs = draws 31 ng n+ , let xs = draws 32 4 n+ ]+ -- A group id absent from rowToGroup reports zero.+ ++ [groupedCase "empty group" [0, 0, 2] 3 (DI.fromList [5, 5, 6 :: Int]) [1, 0, 1]]++-- The public API: aggregate after groupBy dispatches to the kernel.+viaAggregate :: Test+viaAggregate =+ let n = 250000+ gs = draws 41 37 n+ xs = draws 42 5000 n+ df = D.fromColumns [("g", DI.fromList gs), ("x", DI.fromList xs)]+ expected =+ D.fromColumns+ [ ("g", DI.fromList [0 .. 36 :: Int])+ , ("u", DI.fromList (refGrouped gs xs 37))+ ]+ in TestCase+ ( assertEqual+ "aggregate countDistinct"+ expected+ ( df+ & D.groupBy ["g"]+ & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"]+ & D.sortBy [D.Asc (F.col @Int "g")]+ )+ )++-- An Int32 key takes the direct grouping path and must match the reference.+viaAggregateInt32Key :: Test+viaAggregateInt32Key =+ let n = 250000+ gs = draws 43 37 n+ xs = draws 44 5000 n+ df =+ D.fromColumns+ [("g", DI.fromList (map fromIntegral gs :: [Int32])), ("x", DI.fromList xs)]+ expected =+ D.fromColumns+ [ ("g", DI.fromList [0 .. 36 :: Int32])+ , ("u", DI.fromList (refGrouped gs xs 37))+ ]+ in TestCase+ ( assertEqual+ "aggregate countDistinct, Int32 key"+ expected+ ( df+ & D.groupBy ["g"]+ & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"]+ & D.sortBy [D.Asc (F.col @Int32 "g")]+ )+ )++tests :: [Test]+tests =+ map+ (TestLabel "countDistinct")+ ( intGlobal+ ++ [intSentinel, doubles, viaAggregate, viaAggregateInt32Key]+ ++ narrowInts+ ++ texts+ ++ grouped+ )
tests/Operations/Derive.hs view
@@ -23,7 +23,7 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values deriveWAI :: Test deriveWAI =
+ tests/Operations/ExprParse.hs view
@@ -0,0 +1,85 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | Reloading expressions from their two printed forms.+module Operations.ExprParse (tests) where++import qualified Data.Map.Strict as M+import qualified Data.Text as T+import qualified Data.Vector.Unboxed as VU+import qualified DataFrame as D+import DataFrame.Expression.Parse+import qualified DataFrame.Functions as F+import DataFrame.Internal.Expression (Expr, eqExpr, prettyPrint)+import DataFrame.Internal.Expression.Operators ((./=.), (.<=.))+import Test.HUnit++frame :: D.DataFrame+frame =+ D.fromColumns+ [ ("Age", D.fromList [25, 41, 63, 18 :: Int])+ , ("Departure/Arrival time convenient", D.fromList [0, 3, 5, 2 :: Int])+ , ("Class", D.fromList ["Business", "Eco", "Eco Plus", "Eco" :: T.Text])+ , ("Arrival Delay in Minutes", D.fromList [0, 12.5, 3, 0 :: Double])+ ]++-- A fitted-tree shape: cast threshold, Text inequality, operator-char name.+model :: Expr Double+model =+ F.lit 0.1+ * F.ifThenElse+ (F.toDouble (F.col @Int "Age") .<=. F.lit 40.5)+ ( F.ifThenElse+ (F.col @T.Text "Class" ./=. F.lit "Business")+ (F.lit (-0.859))+ (F.lit 0.31)+ )+ ( F.ifThenElse+ (F.toDouble (F.col @Int "Departure/Arrival time convenient") .<=. F.lit 2.5)+ (F.lit 1.2)+ (F.col @Double "Arrival Delay in Minutes" / F.lit 10)+ )+ + F.lit 0.1+ * F.ifThenElse+ (F.col @T.Text "Class" ./=. F.lit "Eco")+ (F.lit 0.5)+ (F.lit (-0.25))++scores :: Expr Double -> VU.Vector Double+scores e =+ either+ (error . show)+ id+ (D.columnAsUnboxedVector (F.col @Double "s") (D.derive "s" e frame))++testShowRoundTrip :: Test+testShowRoundTrip = TestCase $ case parseShownAs @Double (show model) of+ Left err -> assertFailure ("parseShown failed: " ++ err)+ Right e -> do+ assertBool "structurally equal" (eqExpr e model)+ assertEqual "same scores" (scores model) (scores e)++testPrettyRoundTrip :: Test+testPrettyRoundTrip = TestCase $ case parsePrettyAs @Double (schemaOf frame) (prettyPrint model) of+ Left err -> assertFailure ("parsePretty failed: " ++ err)+ Right e -> do+ assertEqual "reprint identical" (prettyPrint model) (prettyPrint e)+ let gap = VU.maximum (VU.zipWith (\a b -> abs (a - b)) (scores model) (scores e))+ assertBool ("scores within 1e-12, got " ++ show gap) (gap < 1e-12)++testSchema :: Test+testSchema = TestCase $ do+ let s = schemaOf frame+ assertEqual "Int column" (Just TInt) (M.lookup "Age" s)+ assertEqual "Text column" (Just TText) (M.lookup "Class" s)+ assertEqual+ "Double column"+ (Just TDouble)+ (M.lookup "Arrival Delay in Minutes" s)++tests :: [Test]+tests =+ [ TestLabel "expr parse: show form round-trips" testShowRoundTrip+ , TestLabel "expr parse: pretty form round-trips" testPrettyRoundTrip+ , TestLabel "expr parse: schema types" testSchema+ ]
tests/Operations/Filter.hs view
@@ -25,7 +25,7 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values filterColumnDoesNotExist :: Test filterColumnDoesNotExist =@@ -86,10 +86,10 @@ TestCase ( assertEqual "Filters out Nothing and unwraps Maybe"- (D.fromNamedColumns [("test", D.fromList $ replicate 5 (1 :: Int))])+ (D.fromColumns [("test", D.fromList $ replicate 5 (1 :: Int))]) ( D.filterJust "test"- ( D.fromNamedColumns+ ( D.fromColumns [("test", D.fromList $ take 10 $ cycle [Just (1 :: Int), Nothing])] ) )
tests/Operations/GroupBy.hs view
@@ -23,7 +23,7 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values groupBySingleRowWAI :: Test groupBySingleRowWAI =@@ -54,29 +54,21 @@ (print $ D.groupBy ["test0"] testData) ) -{- | The grouping invariant every backend (sequential now, parallel in Phase 2)-must satisfy: 'valueIndices' / 'offsets' / 'rowToGroup' agree, groups partition-all rows exactly once, each group is exactly the rows sharing a key tuple, and-no two groups share a key. Checked on a frame with several keys colliding under-the hash (forces the open-addressing key re-verification path).--}+-- | Every backend must partition rows into groups by key tuple. groupingInvariantHolds :: Test groupingInvariantHolds = TestCase (assertBool "grouping partitions rows by exact key tuple" (checkGrouping df)) where- -- Repeated, interleaved keys across two columns; the dense int grid is the- -- classic hash-collision stressor.+ -- Repeated, interleaved keys across two columns stress hash collisions. df =- D.fromNamedColumns+ D.fromColumns [ ("k1", DI.fromList (map (`mod` 5) [0 .. 199 :: Int])) , ("k2", DI.fromList (map (\i -> (i * 7) `mod` 3) [0 .. 199 :: Int])) , ("v", DI.fromList [0 .. 199 :: Int]) ] -{- | Recompute the reference grouping by the key tuple and assert the library's-'valueIndices'/'offsets'/'rowToGroup' describe the same partition.--}+-- | Recompute the reference grouping by key tuple and compare the partitions. checkGrouping :: D.DataFrame -> Bool checkGrouping df = let g = D.groupBy ["k1", "k2"] df@@ -91,13 +83,13 @@ Nothing -> error "missing column" key i = (k1 !! i, k2 !! i) -- The group slices read out of vis/os.- slices = [take (e - s) (drop s vis) | (s, e) <- zip os (tail os)]+ slices = [take (e - s) (drop s vis) | (s, e) <- zip os (drop 1 os)] -- valueIndices must be a permutation of all rows. permutationOk = L.sort vis == [0 .. n - 1] -- Every slice is non-empty and internally key-constant. constKeys = all (\sl -> not (L.null sl) && allSame (map key sl)) slices -- Distinct groups have distinct keys.- groupKeys = map (key . head) slices+ groupKeys = [key s | (s : _) <- slices] distinctKeys = L.length (L.nub groupKeys) == L.length groupKeys -- The number of groups equals the number of distinct key tuples. nGroupsOk = L.length slices == M.size (M.fromList [(key i, ()) | i <- [0 .. n - 1]])
tests/Operations/Inference.hs view
@@ -1,11 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Round-2 (S2) pins for the single-pass inference lattice and the-Int -> Double promotion path: sample classification via the WS-B byte-parsers, prefix promotion instead of full-column re-parse, and the-byte-level missing-token test in the default reader's inferred path.--}+-- | Pins for the single-pass inference lattice and Int to Double promotion. module Operations.Inference where import qualified Data.ByteString as BS@@ -29,9 +25,7 @@ assumeText :: [Maybe T.Text] -> D.ParsingAssumption assumeText = D.makeParsingAssumption "%Y-%m-%d" . V.fromList --- The candidate-mask priorities reproduce the documented fallback order:--- Bool, then Int (only when the Double mask agrees), Double, Date, Text;--- an all-null sample stays NoAssumption.+-- Candidate-mask priorities reproduce the documented fallback order. latticePriorities :: Test latticePriorities = TestCase $ do let cases =@@ -55,9 +49,7 @@ ) cases --- Pinned new behavior (WS-B parser semantics in the sample): an--- overflowing integer clears the Int candidate instead of wrapping, so--- the column classifies (and parses) as Double.+-- An overflowing integer clears the Int candidate, so the column is Double. latticeOverflowIntIsDouble :: Test latticeOverflowIntIsDouble = TestCase $ do assertEqual@@ -69,17 +61,13 @@ D.DoubleAssumption (assumeText [Just "9223372036854775808"]) --- Pinned new behavior: byte-level strip-tolerant classification accepts--- ASCII-padded numerics (the strip-equivalent unification, audit--- divergence #2), where the old Text-path sample rejected " 42" as--- Double and demoted to Text.+-- Byte-level strip-tolerant classification accepts ASCII-padded numerics. latticePaddedIntIsInt :: Test latticePaddedIntIsInt = TestCase (assertEqual "padded int" D.IntAssumption (assumeText [Just " 42 "])) --- End-to-end: an overflowing cell inside the sample now yields a Double--- column with the correctly converted value (old: silently wrapped Int).+-- An overflowing cell inside the sample yields a Double column. parseOverflowIntsAsDoubles :: Test parseOverflowIntsAsDoubles = let beforeParse = ["9223372036854775808", "1", "2"] :: [T.Text]@@ -90,8 +78,7 @@ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "overflow -> Double column" expected actual) --- Promotion: a Double cell far past the sample converts the built Int--- prefix in place; values must equal a from-scratch Double parse.+-- A Double cell far past the sample promotes the built Int prefix in place. promoteIntPrefixToDouble :: Test promoteIntPrefixToDouble = let ints = map (T.pack . show) ([1 .. 150] :: [Int])@@ -104,8 +91,7 @@ $ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "Int prefix promoted to Double" expected actual) --- An overflowing integer past the sample triggers promotion with its--- true Double value (the old Text-path Int parse silently wrapped it).+-- An overflowing integer past the sample promotes with its true value. promoteOverflowPastSample :: Test promoteOverflowPastSample = let ints = map (T.pack . show) ([1 .. 30] :: [Int])@@ -141,8 +127,7 @@ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "nulls survive promotion" expected actual) --- A cell that parses as neither Int nor Double demotes the whole column--- to Text with every raw cell preserved (re-extracted, not re-parsed).+-- A cell parsing as neither Int nor Double demotes the column to Text. promotionDemotesToText :: Test promotionDemotesToText = let beforeParse = map (T.pack . show) ([1 .. 30] :: [Int]) ++ ["abc"]@@ -153,8 +138,7 @@ $ DI.fromVector (V.fromList beforeParse) in TestCase (assertEqual "demotes to Text, raw values kept" expected actual) --- Default (cassava) reader, inferred path: the same promotion runs over--- the retained raw bytes.+-- Default (cassava) reader, inferred path: promotion runs over raw bytes. readerPromotesIntColumn :: Test readerPromotesIntColumn = TestCase $ do let csv =@@ -170,8 +154,7 @@ Just col -> assertEqual "reader Int->Double promotion" expected col Nothing -> assertFailure "column x missing" --- Default reader: canonical missing tokens null the cell through the--- byte-level fast path, exactly as the Text-decode check did.+-- Default reader: canonical missing tokens null the cell via the fast path. readerCanonicalMissingTokens :: Test readerCanonicalMissingTokens = TestCase $ do let csv = "x,y\nNA,1\nnan,2\nNothing,3\n ,4\nNULL,5\n"@@ -185,8 +168,7 @@ Just col -> assertEqual "int column" expectedY col Nothing -> assertFailure "column y missing" --- A custom missing list replaces (not extends) the canonical one: "NA"--- must then survive as text.+-- A custom missing list replaces the canonical one, so "NA" stays text. readerCustomMissingTokens :: Test readerCustomMissingTokens = TestCase $ do let csv = "x\nfoo\nNA\n7\n"@@ -199,8 +181,7 @@ Just col -> assertEqual "custom list only" expected col Nothing -> assertFailure "column x missing" --- Default reader date columns still parse (the WS-B fast date path is--- bit-identical to readByteStringDate for %Y-%m-%d).+-- Default reader date columns still parse via the fast date path. readerDatesStillParse :: Test readerDatesStillParse = TestCase $ do let csv = "d\n2024-01-01\n2024-01-02\n2024-01-03\n"
tests/Operations/InsertColumn.hs view
@@ -14,7 +14,7 @@ testData :: D.DataFrame testData =- D.fromNamedColumns+ D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) , ("test3", DI.fromList ([1 .. 26] :: [Int]))@@ -25,7 +25,7 @@ , ("test8", DI.fromList ['a' .. 'z']) ] --- Adding a boxed vector to an empty dataframe creates a new column boxed containing the vector elements.+-- Adding a boxed vector to an empty dataframe creates a boxed column. addBoxedColumn :: Test addBoxedColumn = TestCase@@ -53,7 +53,7 @@ ) ) --- Adding an boxed vector with an unboxable type (Int/Double) to an empty dataframe creates a new column boxed containing the vector elements.+-- Adding a boxed vector of an unboxable type creates a boxed column. addUnboxedColumn :: Test addUnboxedColumn = TestCase@@ -72,7 +72,7 @@ (DI.getColumn "new" $ D.insertColumn "new" (DI.fromList [1 :: Int, 2, 3]) D.empty) ) --- Adding a column with less values than the current DF dimensions adds column with optionals.+-- Adding a column with fewer values than the frame adds optionals. addSmallerColumnBoxed :: Test addSmallerColumnBoxed = TestCase@@ -132,7 +132,7 @@ TestCase ( assertEqual "Smaller lists should grow and contain optionals"- ( D.fromNamedColumns+ ( D.fromColumns [ ( "new" , D.fromList [Just "a" :: Maybe T.Text, Just "b", Just "c", Nothing, Nothing]@@ -150,7 +150,7 @@ TestCase ( assertEqual "Smaller lists should grow and contain optionals"- ( D.fromNamedColumns+ ( D.fromColumns [ ("old", D.fromList [Just 1 :: Maybe Int, Just 2, Nothing, Nothing, Nothing]) , ("new", D.fromList [Just 1 :: Maybe Int, Just 2, Just 3, Nothing, Nothing]) , ("newer", D.fromList [1 :: Int, 2, 3, 4, 5])
+ tests/Operations/InterpreterKernels.hs view
@@ -0,0 +1,202 @@+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-}++-- | Kernels agree with plain Haskell row by row; floats compare bitwise.+module Operations.InterpreterKernels where++import Control.Exception (throw)+import qualified Data.Text as T+import qualified Data.Vector as V++import qualified DataFrame as D+import DataFrame.Expression.Operators (+ (./=.),+ (.<.),+ (.<=.),+ (.==.),+ (.>.),+ (.>=.),+ )+import qualified DataFrame.Functions as F+import DataFrame.Internal.Column (+ Column,+ Columnable,+ TypedColumn (..),+ toVector,+ )+import DataFrame.Internal.Expression (Expr (..), prettyPrint)+import DataFrame.Internal.Interpreter (interpret)++import Assertions (BitEq, assertBitEq)+import Test.HUnit++-- | One side of an expression: a column with known rows, or a literal.+data Operand e = Column T.Text [e] | Literal e++operandExpr :: (Columnable e) => Operand e -> Expr e+operandExpr (Column name _) = Col name+operandExpr (Literal v) = Lit v++-- | The operand's value in each of @n@ rows.+operandRows :: Int -> Operand e -> [e]+operandRows _ (Column _ rows) = rows+operandRows n (Literal v) = replicate n v++operandColumns :: (Columnable e) => Operand e -> [(T.Text, Column)]+operandColumns (Column name rows) = [name D.=: rows]+operandColumns (Literal _) = []++-- | An operator paired with what it computes for one row.+data BinOp e r = BinOp (Expr e -> Expr e -> Expr r) (e -> e -> r)++comparisons :: (Columnable e, Ord e) => [BinOp e Bool]+comparisons =+ [ BinOp (.<.) (<)+ , BinOp (.<=.) (<=)+ , BinOp (.>.) (>)+ , BinOp (.>=.) (>=)+ , BinOp (.==.) (==)+ , BinOp (./=.) (/=)+ ]++arithmetic :: (Columnable e, Num e) => [BinOp e e]+arithmetic = [BinOp (+) (+), BinOp (-) (-), BinOp (*) (*)]++doubles :: [Double]+doubles = [0 / 0, -(1 / 0), -1.5, -0.0, 0.0, 5.0e-324, 0.5, 1, 2.5, 1.0e308, 1 / 0]++ints :: [Int]+ints = [minBound, -3, -1, 0, 1, 7, maxBound]++-- | The second size is above the 200k threshold where loops run in parallel.+sizes :: [Int]+sizes = [length doubles * length doubles, 200003]++-- | Two columns holding every pair of the given values, cycled to @n@ rows.+columnPair :: Int -> [e] -> (Operand e, Operand e)+columnPair n vals = (Column "x" xs, Column "y" ys)+ where+ (xs, ys) = unzip (take n (cycle [(a, b) | a <- vals, b <- vals]))++-- | Column-column, column-literal and literal-column operand pairs.+operandPairs :: Int -> [e] -> [(Operand e, Operand e)]+operandPairs n vals = (x, y) : concat [[(x, Literal t), (Literal t, x)] | t <- lits]+ where+ (x, y) = columnPair n vals+ lits = if n > 1000 then take 3 vals else vals++run :: forall a. (Columnable a) => [(T.Text, Column)] -> Expr a -> [a]+run columns e = case interpret @a (D.fromColumns columns) e of+ Left err -> throw err+ Right (TColumn col) -> V.toList (either throw id (toVector @a @V.Vector col))++binaryCases ::+ (Columnable e, Columnable r, BitEq r, Show r) => [e] -> [BinOp e r] -> [Test]+binaryCases vals ops =+ [ TestCase+ (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e))+ | n <- sizes+ , (l, r) <- operandPairs n vals+ , let columns = operandColumns l ++ operandColumns r+ , BinOp op f <- ops+ , let e = op (operandExpr l) (operandExpr r)+ expected = zipWith f (operandRows n l) (operandRows n r)+ ]++-- | If-then-else over a Bool column, with branches of every operand mix.+selectCases :: (Columnable e, BitEq e, Show e) => [e] -> [Test]+selectCases vals =+ [ TestCase+ (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e))+ | n <- sizes+ , let cond = Column "c" (take n (cycle [True, False, False]))+ lits = take 3 vals+ , (l, r) <-+ operandPairs n vals ++ zip (map Literal lits) (map Literal (drop 1 lits))+ , let columns = operandColumns cond ++ operandColumns l ++ operandColumns r+ e = If (operandExpr cond) (operandExpr l) (operandExpr r)+ expected =+ zipWith3+ (\c a b -> if c then a else b)+ (operandRows n cond)+ (operandRows n l)+ (operandRows n r)+ ]++caseWhenCases :: [Test]+caseWhenCases =+ [ TestCase+ (assertBitEq (prettyPrint e ++ " n=" ++ show n) expected (run columns e))+ | n <- sizes+ , let x = fst (columnPair n ints)+ columns = operandColumns x+ xe = operandExpr x+ e =+ F.caseWhen (xe .>. Lit 5) (Lit 1) $+ F.caseWhen (xe .>. Lit 0) (Lit 2) $+ F.orElse (Lit (3 :: Int))+ expected = [if v > 5 then 1 else if v > 0 then 2 else 3 | v <- operandRows n x]+ ]++toDoubleCases :: [Test]+toDoubleCases =+ [ TestCase+ ( assertBitEq+ (prettyPrint e ++ " n=" ++ show n)+ (map fromIntegral (operandRows n x))+ (run (operandColumns x) e)+ )+ | n <- sizes+ , let x = fst (columnPair n ints)+ e = F.toDouble (operandExpr x)+ ]++mergeAggCases :: [Test]+mergeAggCases = concatMap sizeCases sizes+ where+ sizeCases n =+ [ TestCase $+ assertEqual+ ("count over " ++ show n ++ " rows")+ [n]+ (take 1 (run (operandColumns xsI) (F.count (F.col @Int "x"))))+ , TestCase $+ assertEqual+ ("countDistinct of an expression over " ++ show n ++ " rows")+ [length (takeWhile (< n) [0 .. 6])]+ (take 1 (run (operandColumns xsI) (F.countDistinct (F.col @Int "x" + 1))))+ , TestCase $+ assertBool+ ("mean of ints over " ++ show n ++ " rows")+ ( all+ (approx (sumD / fromIntegral n))+ (run (operandColumns xsI) (F.mean (F.col @Int "x")))+ )+ , TestCase $+ assertEqual+ ("mean of doubles over " ++ show n ++ " rows")+ [sum (take n smallDoubles) / fromIntegral n]+ (take 1 (run (operandColumns xsD) (F.mean (F.col @Double "x"))))+ ]+ where+ -- Seven distinct ints with a large spread; the sum fits in a Double.+ xs = take n (cycle [3, -1, 0, 1, 7, 1000000007, -999999989 :: Int])+ xsI = Column "x" xs+ xsD = Column "x" (take n smallDoubles)+ sumD = sum (map fromIntegral xs) :: Double+ smallDoubles = cycle [0.5, 1, 2.5, -3, 0.25]+ approx a b = abs (a - b) <= 1.0e-9 * max 1 (abs a)++tests :: [Test]+tests =+ mergeAggCases+ ++ binaryCases doubles comparisons+ ++ binaryCases ints comparisons+ ++ binaryCases doubles arithmetic+ ++ binaryCases ints arithmetic+ ++ selectCases doubles+ ++ selectCases ints+ ++ caseWhenCases+ ++ toDoubleCases
tests/Operations/Join.hs view
@@ -16,14 +16,14 @@ df1 :: D.DataFrame df1 =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) ] df2 :: D.DataFrame df2 =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) ]@@ -48,7 +48,7 @@ TestCase ( assertEqual "Test inner join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"])@@ -62,7 +62,7 @@ TestCase ( assertEqual "Test left join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"])@@ -76,7 +76,7 @@ TestCase ( assertEqual "Test right join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList [Just "A0" :: Maybe Text, Just "A1", Just "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"])@@ -96,7 +96,7 @@ TestCase ( assertEqual "Test typed inner join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"])@@ -110,7 +110,7 @@ TestCase ( assertEqual "Test typed left join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"])@@ -122,7 +122,7 @@ -- A right-hand frame whose payload column is already optional. dfOptional :: D.DataFrame dfOptional =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1"]) , ("C", D.fromList [Just 10 :: Maybe Int, Just 11]) ]@@ -131,16 +131,13 @@ DT.TypedDataFrame ['("key", Text), '("C", Maybe Int)] tdfOptional = either (error . show) id (DT.freezeWithError dfOptional) -{- | A left join over an already-optional column must not nest the Maybe: the-explicit @Maybe Int@ result schema below only type-checks because 'WrapMaybe'-flattens @Maybe (Maybe Int)@ to @Maybe Int@, matching the runtime column.--}+-- | A left join over an already-optional column must not nest the Maybe. testLeftJoinTypedOptional :: Test testLeftJoinTypedOptional = TestCase ( assertEqual "Typed left join keeps an already-optional column single-Maybe"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3", "K4", "K5"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2", "A3", "A4", "A5"]) ,@@ -163,7 +160,7 @@ TestCase ( assertEqual "Test typed right join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("A", D.fromList [Just "A0" :: Maybe Text, Just "A1", Just "A2"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"])@@ -195,7 +192,7 @@ TestCase ( assertEqual "Test full outer join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ( "Name" , D.fromList ["James" :: Text, "Kelly", "Mike", "Sally"]@@ -224,7 +221,7 @@ TestCase ( assertEqual "Full outer join on an unboxed (Int) key column coalesces correctly"- ( D.fromNamedColumns+ ( D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 3, 4]) , ( "amount"@@ -244,21 +241,21 @@ ordersDf :: D.DataFrame ordersDf =- D.fromNamedColumns+ D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 4]) , ("amount", D.fromList [250.0 :: Double, 80.0, 125.0]) ] customersDf :: D.DataFrame customersDf =- D.fromNamedColumns+ D.fromColumns [ ("customer_id", D.fromList [1 :: Int, 2, 3]) , ("name", D.fromList ["Ada" :: Text, "Lin", "Grace"]) ] dfL :: D.DataFrame dfL =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ("X", D.fromList ["LX0" :: Text, "LX1", "LX2"]) , ("Lonly", D.fromList ["L0" :: Text, "L1", "L2"])@@ -266,7 +263,7 @@ dfR :: D.DataFrame dfR =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K3"]) , ("X", D.fromList ["RX0" :: Text, "RX1", "RX3"]) , ("Ronly", D.fromList [10 :: Int, 11, 13])@@ -277,7 +274,7 @@ TestCase ( assertEqual "Test inner join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1"]) , ("X", D.fromList [These "LX0" "RX0" :: These Text Text, These "LX1" "RX1"]) , ("Lonly", D.fromList ["L0" :: Text, "L1"])@@ -292,7 +289,7 @@ TestCase ( assertEqual "Test left join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2"]) , ( "X"@@ -310,7 +307,7 @@ TestCase ( assertEqual "Test right join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K3"]) , ( "X"@@ -328,7 +325,7 @@ TestCase ( assertEqual "Test right join with single key"- ( D.fromNamedColumns+ ( D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K2", "K3"]) , ( "X"@@ -390,7 +387,7 @@ testInnerJoinMissingKeysSuggestion = TestCase $ let typoDf =- D.fromNamedColumns+ D.fromColumns [ ("hello", D.fromList ["H" :: Text]) , ("world", D.fromList ["W" :: Text]) ]@@ -402,14 +399,14 @@ -- Empty DataFrame fixtures: same schema as df1/df2 but zero rows. emptyDf1 :: D.DataFrame emptyDf1 =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ([] :: [Text])) , ("A", D.fromList ([] :: [Text])) ] emptyDf2 :: D.DataFrame emptyDf2 =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ([] :: [Text])) , ("B", D.fromList ([] :: [Text])) ]@@ -451,19 +448,16 @@ ) -- Many-to-many: duplicate keys on both sides produce the cross-product.--- manyLeft: K0->A0, K1->A1, K1->A2--- manyRight: K0->B0, K1->B1, K1->B2--- Expected inner join: 1 K0 pair + 4 K1 pairs = 5 rows manyLeft :: D.DataFrame manyLeft =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K1"]) , ("A", D.fromList ["A0" :: Text, "A1", "A2"]) ] manyRight :: D.DataFrame manyRight =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList ["K0" :: Text, "K1", "K1"]) , ("B", D.fromList ["B0" :: Text, "B1", "B2"]) ]@@ -487,17 +481,16 @@ ) -- Stress the open-addressing hash index with many distinct integer keys.--- Left keys 0..9999, right keys 5000..14999 (overlap 5000..9999). bigLeft :: D.DataFrame bigLeft =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList [0 .. 9999 :: Int]) , ("la", D.fromList [i * 2 | i <- [0 .. 9999 :: Int]]) ] bigRight :: D.DataFrame bigRight =- D.fromNamedColumns+ D.fromColumns [ ("key", D.fromList [5000 .. 14999 :: Int]) , ("rb", D.fromList [i * 3 | i <- [5000 .. 14999 :: Int]]) ]
tests/Operations/Merge.hs view
@@ -9,32 +9,32 @@ df1 :: D.DataFrame df1 =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList [1 :: Int, 2]) , ("B", DI.fromList ['a', 'b']) ] df2 :: D.DataFrame df2 =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList [3 :: Int]) , ("B", DI.fromList ['c']) ] dfOnlyA :: D.DataFrame-dfOnlyA = D.fromNamedColumns [("A", DI.fromList [1 :: Int, 2])]+dfOnlyA = D.fromColumns [("A", DI.fromList [1 :: Int, 2])] dfOnlyB :: D.DataFrame-dfOnlyB = D.fromNamedColumns [("B", DI.fromList ['c', 'd'])]+dfOnlyB = D.fromColumns [("B", DI.fromList ['c', 'd'])] dfSide1 :: D.DataFrame dfSide1 =- D.fromNamedColumns+ D.fromColumns [("L", DI.fromList [10 :: Int, 20])] dfSide2 :: D.DataFrame dfSide2 =- D.fromNamedColumns+ D.fromColumns [("R", DI.fromList ["x" :: String, "y"])] mergeVerticalWAI :: Test@@ -42,7 +42,7 @@ TestCase ( assertEqual "Vertical merge concatenates matching columns"- ( D.fromNamedColumns+ ( D.fromColumns [ ("A", DI.fromList [1 :: Int, 2, 3]) , ("B", DI.fromList ['a', 'b', 'c']) ]@@ -55,7 +55,7 @@ TestCase ( assertEqual "Disjoint schemas produce optional padding"- ( D.fromNamedColumns+ ( D.fromColumns [ ( "A" , DI.fromList@@ -102,7 +102,7 @@ TestCase ( assertEqual "Horizontal merge combines columns side by side"- ( D.fromNamedColumns+ ( D.fromColumns [ ("L", DI.fromList [10 :: Int, 20]) , ("R", DI.fromList ["x" :: String, "y"]) ]@@ -131,14 +131,14 @@ mergeVerticalDifferentTypesSameColumnName :: Test mergeVerticalDifferentTypesSameColumnName = let- dfIntA = D.fromNamedColumns [("A", DI.fromList [1 :: Int, 2])]+ dfIntA = D.fromColumns [("A", DI.fromList [1 :: Int, 2])] - dfTextA = D.fromNamedColumns [("A", DI.fromList ["x" :: String, "y"])]+ dfTextA = D.fromColumns [("A", DI.fromList ["x" :: String, "y"])] in TestCase ( assertEqual "Merging columns with same name but different types wraps the values in Either"- ( D.fromNamedColumns+ ( D.fromColumns [ ("A", DI.fromList [Left (1 :: Int), Left 2, Right ("x" :: String), Right "y"]) ] )
+ tests/Operations/MonomorphicLoops.hs view
@@ -0,0 +1,324 @@+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-}++-- | Per-type column loops checked against list references and null handling.+module Operations.MonomorphicLoops where++import Control.Exception (throw)+import qualified Data.Text as T+import qualified Data.Vector as V+import qualified Data.Vector.Unboxed as VU+import Data.Word (Word64)+import GHC.Float (castDoubleToWord64, castFloatToWord32)++import DataFrame.Internal.Column (+ Column,+ Columnable,+ findIndices,+ foldLinearGroups,+ foldl1Column,+ foldl1DirectGroups,+ foldlColumn,+ fromList,+ fromUnboxedVector,+ headColumn,+ ifoldrColumn,+ imapColumn,+ mapColumn,+ toVector,+ zipColumns,+ zipWithColumns,+ )++import Test.HUnit++sizes :: [Int]+sizes = [37, 200003]++doublesOf :: Int -> [Double]+doublesOf n = take n (cycle [0 / 0, -(1 / 0), -1.5, -0.0, 0.0, 5.0e-324, 0.5, 2.5, 1 / 0])++intsOf :: Int -> [Int]+intsOf n = take n (cycle [minBound, -3, 0, 1, 7, maxBound])++boolsOf :: Int -> [Bool]+boolsOf n = take n (cycle [True, False, False])++col :: (Columnable a, VU.Unbox a) => [a] -> Column+col = fromUnboxedVector . VU.fromList++out :: forall a e. (Columnable a) => Either e Column -> [a]+out (Left _) = error "column op failed"+out (Right c) = V.toList (either throw id (toVector @a @V.Vector c))++-- | Element results compare by bits for floating types, by '==' otherwise.+class Same a where+ same :: [a] -> [a] -> Bool++instance Same Double where+ same a b = map castDoubleToWord64 a == map castDoubleToWord64 b+instance Same Int where+ same = (==)+instance Same Bool where+ same = (==)+instance Same Float where+ same a b = map castFloatToWord32 a == map castFloatToWord32 b++bits :: [Double] -> [Word64]+bits = map castDoubleToWord64++-- | map, imap and zip of one (element, result) pairing.+pairCases ::+ forall a c.+ (Columnable a, VU.Unbox a, Columnable c, Same c) =>+ String -> (Int -> [a]) -> (a -> c) -> (Int -> a -> c) -> (a -> a -> c) -> [Test]+pairCases label gen f fi f2 =+ [ TestList+ [ TestCase+ ( assertBool+ (label ++ " map n=" ++ show n)+ (same (out @c (mapColumn f (col xs))) (map f xs))+ )+ , TestCase+ ( assertBool+ (label ++ " imap n=" ++ show n)+ (same (out @c (imapColumn fi (col xs))) (zipWith fi [0 ..] xs))+ )+ , TestCase+ ( assertBool+ (label ++ " zip n=" ++ show n)+ (same (out @c (zipWithColumns f2 (col xs) (col ys))) (zipWith f2 xs ys))+ )+ ]+ | n <- sizes+ , let xs = gen n+ ys = reverse xs+ ]++foldCases :: [Test]+foldCases =+ concat+ [ [ TestCase+ ( assertEqual+ ("foldl DD n=" ++ show n)+ (bits [foldl (+) 0.25 ds])+ (bits [fold (+) 0.25 ds])+ )+ , TestCase+ ( assertEqual+ ("foldl DI n=" ++ show n)+ (foldl (\k d -> k + fromEnum (d > 0)) 0 ds)+ (fold (\k d -> k + fromEnum (d > 0)) 0 ds)+ )+ , TestCase+ ( assertEqual+ ("foldl ID n=" ++ show n)+ (bits [foldl (\a i -> a + fromIntegral i) 0 is])+ (bits [fold (\a i -> a + fromIntegral i) (0 :: Double) is])+ )+ , TestCase+ (assertEqual ("foldl II n=" ++ show n) (foldl (+) 1 is) (fold (+) 1 is))+ , TestCase+ ( assertEqual+ ("foldl1 D n=" ++ show n)+ (bits [foldl1 (-) ds])+ (bits [fold1 (-) ds])+ )+ , TestCase (assertEqual ("foldl1 I n=" ++ show n) (maximum is) (fold1 max is))+ , TestCase+ ( assertEqual+ ("foldl Float n=" ++ show n)+ (castFloatToWord32 (sum fs))+ (castFloatToWord32 (fold (+) (0 :: Float) fs))+ )+ ]+ | n <- sizes+ , let ds = doublesOf n+ is = intsOf n+ fs = map realToFrac ds :: [Float]+ ]+ where+ fold ::+ (Columnable a, VU.Unbox a, Columnable b) => (b -> a -> b) -> b -> [a] -> b+ fold f z xs = either throw id (foldlColumn f z (col xs))+ fold1 :: (Columnable a, VU.Unbox a) => (a -> a -> a) -> [a] -> a+ fold1 f xs = either throw id (foldl1Column f (col xs))++-- | Mapping at the element type over a nullable column keeps its bitmap.+nullableCases :: [Test]+nullableCases =+ [ TestCase+ ( assertEqual+ ("nullable map n=" ++ show n)+ (map (fmap (castDoubleToWord64 . (* 2))) xs)+ ( map+ (fmap castDoubleToWord64)+ (out @(Maybe Double) (mapColumn @Double (* 2) (fromList xs)))+ )+ )+ | n <- sizes+ , let xs =+ take n (cycle [Just 1.5, Nothing, Just (-0.0), Just (0 / 0)]) :: [Maybe Double]+ ]++-- Functions over a column with nulls: f never runs on a null row.+nullSemantics :: [Test]+nullSemantics =+ [ eq+ "map skips null rows"+ [Just 25, Nothing, Just 50]+ (outM @Int (mapColumn @Int (100 `div`) ints3))+ , eq+ "map from boxed nullable"+ [Just 2, Nothing, Just 0]+ (outM @Int (mapColumn @T.Text T.length texts3))+ , eq+ "map to Maybe"+ [Just 4, Nothing, Nothing]+ (outM @Int (mapColumn @Int keepBig ints3))+ , eq+ "imap skips null rows"+ [Just 4, Nothing, Just 4]+ (outM @Int (imapColumn @Int (flip (+)) ints3))+ , eq+ "zip: null if either side is"+ [Just 5, Nothing, Nothing]+ (outM @Int (zipWithColumns @Int @Int (\a b -> a + b `div` b) ints3 other3))+ , eq+ "zip to Maybe"+ [Nothing, Nothing, Just 3]+ ( outM @Int+ ( zipWithColumns @Int @Int+ (\a b -> if a < b then Just (b - a) else Nothing)+ ints3+ other3'+ )+ )+ , eq "foldl skips null rows" (Just 6) (ok $ foldlColumn @Int (+) 0 ints3)+ , eq "foldl1 skips null rows" (Just 4) (ok $ foldl1Column @Int max ints3)+ , TestCase+ ( assertBool+ "foldl1 over only nulls fails"+ ( either+ (const True)+ (const False)+ (foldl1Column @Int (+) (fromList [Nothing, Nothing :: Maybe Int]))+ )+ )+ , eq+ "ifoldr skips null rows"+ (Just [0, 2])+ (ok $ ifoldrColumn @Int (\i _ r -> i : r) [] ints3)+ , eq+ "findIndices skips null rows"+ (Just (VU.fromList [0]))+ (ok $ findIndices @Int (== 4) (fromList [Just 4, Nothing, Just (1 :: Int)]))+ , eq+ "findIndices never matches a null"+ (Just VU.empty)+ (ok $ findIndices @Int (== 0) (fromList [Just 3, Nothing :: Maybe Int]))+ , eq+ "head is the first valid row"+ (Just 3)+ (ok $ headColumn @Int (fromList [Nothing, Just (3 :: Int)]))+ , eq+ "zipColumns keeps nulls"+ [Just (4, 1), Nothing, Nothing]+ (outM @(Int, Int) (Right (zipColumns ints3 other3)))+ , eq+ "foldLinearGroups skips null rows"+ [1, 2]+ ( out @Int+ ( foldLinearGroups @Int+ (+)+ 0+ (fromList [Just 1, Nothing, Just 2, Nothing :: Maybe Int])+ (VU.fromList [0, 0, 1, 1])+ 2+ )+ )+ , eq+ "foldl1DirectGroups: all-null group is null"+ [Nothing, Just 7]+ ( outM @Int+ ( foldl1DirectGroups @Int+ (+)+ (fromList [Nothing, Nothing, Just 3, Just (4 :: Int)])+ (VU.fromList [0 .. 3])+ (VU.fromList [0, 2, 4])+ )+ )+ , TestCase+ ( assertBool+ "toVector refuses an element read with nulls"+ (either (const True) (const False) (toVector @Int @V.Vector ints3))+ )+ , eq+ "toVector allows a bitmap with no nulls"+ (Just [1, 2])+ (ok $ V.toList <$> toVector @Int @V.Vector (fromList [Just 1, Just (2 :: Int)]))+ ]+ where+ ok :: Either e x -> Maybe x+ ok = either (const Nothing) Just+ eq :: (Eq x, Show x) => String -> x -> x -> Test+ eq label expected actual = TestCase (assertEqual label expected actual)+ ints3 = fromList [Just 4, Nothing, Just (2 :: Int)]+ other3 = fromList [Just 1, Just 0, Nothing :: Maybe Int]+ other3' = fromList [Just 1, Just 0, Just (5 :: Int)]+ texts3 = fromList [Just ("ab" :: T.Text), Nothing, Just ""]+ keepBig x = if x > 3 then Just x else Nothing++outM :: forall a e. (Columnable a) => Either e Column -> [Maybe a]+outM (Left _) = error "column op failed"+outM (Right c) = V.toList (either throw id (toVector @(Maybe a) @V.Vector c))++tests :: [Test]+tests =+ concat+ [ nullableCases+ , nullSemantics+ , pairCases @Double @Double+ "DD"+ doublesOf+ (* 1.5)+ (\i d -> d + fromIntegral i)+ (-)+ , pairCases @Double @Int+ "DI"+ doublesOf+ (fromEnum . (> 0))+ (\i d -> i + fromEnum (d < 0))+ (\a b -> fromEnum (a <= b))+ , pairCases @Double @Bool "DB" doublesOf (<= 0.5) (\i d -> even i || d > 0) (<)+ , pairCases @Int @Double+ "ID"+ intsOf+ fromIntegral+ (\i x -> fromIntegral (x + i))+ (\a b -> fromIntegral a - fromIntegral b)+ , pairCases @Int @Int "II" intsOf (* 3) (+) (-)+ , pairCases @Int @Bool "IB" intsOf odd (flip (>)) (==)+ , pairCases @Bool @Double+ "BD"+ boolsOf+ (\b -> if b then 2.5 else 0 / 0)+ (\i b -> if b then fromIntegral i else -0.0)+ (\a b -> if a && b then 1 else -1)+ , pairCases @Bool @Int+ "BI"+ boolsOf+ fromEnum+ (\i b -> if b then i else -i)+ (\a b -> fromEnum a + fromEnum b)+ , pairCases @Bool @Bool "BB" boolsOf not (\i b -> b /= even i) (/=)+ , pairCases @Float @Float+ "Float fallback"+ (map realToFrac . doublesOf)+ (* 2)+ (\i x -> x + fromIntegral i)+ (+)+ , foldCases+ ]
tests/Operations/Nullable.hs view
@@ -20,14 +20,10 @@ import Test.HUnit --- ------------------------------------------------------------------------------ Untyped Expr layer tests--- ---------------------------------------------------------------------------- -- DataFrame with one plain Int column and one Maybe Int column testData :: D.DataFrame testData =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1, 2, 3] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 10, Nothing, Just 30 :: Maybe Int])) ]@@ -141,10 +137,6 @@ ) ) --- ------------------------------------------------------------------------------ nullLift / nullLift2 — untyped layer--- ---------------------------------------------------------------------------- -- | nullLift negate on Maybe Int propagates Nothing nullLiftMaybeInt :: Test nullLiftMaybeInt =@@ -222,14 +214,10 @@ ) ) --- ------------------------------------------------------------------------------ Cross-type arithmetic (Int × Double promotion)--- ---------------------------------------------------------------------------- -- DataFrame with Int, Maybe Int, Double, Maybe Double columns crossData :: D.DataFrame crossData =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1, 2, 3] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 10, Nothing, Just 30 :: Maybe Int])) , ("d", DI.fromList ([1.5, 2.5, 3.5] :: [Double]))@@ -451,7 +439,7 @@ -- Division tests use clean-dividing data: x=[2,4,6], d=[1,2,3] divData :: D.DataFrame divData =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([2, 4, 6] :: [Int])) , ("y", DI.fromVector (V.fromList [Just 4, Nothing, Just 6 :: Maybe Int])) , ("d", DI.fromList ([1.0, 2.0, 3.0] :: [Double]))@@ -527,10 +515,6 @@ ) ) --- ------------------------------------------------------------------------------ Typed TExpr layer — cross-type arithmetic--- ---------------------------------------------------------------------------- type CrossSchema = '[ '("x", Int) , '("y", Maybe Int)@@ -615,10 +599,6 @@ ) ) --- ------------------------------------------------------------------------------ Typed TExpr layer tests--- ---------------------------------------------------------------------------- type TestSchema = '[ '("x", Int), '("y", Maybe Int)] typedTestData :: DT.TypedDataFrame TestSchema@@ -725,10 +705,6 @@ ) ) --- ------------------------------------------------------------------------------ Grouped aggregation over nullable columns--- ---------------------------------------------------------------------------- -- | Interleaved, so grouping's row permutation is not the identity. interleavedKeys :: [T.Text] interleavedKeys = ["a", "b", "a", "b", "a", "b", "a", "b"]@@ -740,7 +716,7 @@ nullsInOneGroup :: D.DataFrame nullsInOneGroup =- D.fromNamedColumns+ D.fromColumns [ ("k", DI.fromList interleavedKeys) , ("v", DI.fromVector (V.fromList nullsInOneGroupValues)) ]@@ -752,7 +728,7 @@ nullsInBothGroups :: D.DataFrame nullsInBothGroups =- D.fromNamedColumns+ D.fromColumns [ ("k", DI.fromList interleavedKeys) , ("v", DI.fromVector (V.fromList nullsInBothGroupsValues)) ]@@ -766,7 +742,7 @@ expectedGroupSums :: [Double] -> D.DataFrame expectedGroupSums sums =- D.fromNamedColumns+ D.fromColumns [ ("k", DI.fromList (["a", "b"] :: [T.Text])) , ("s", DI.fromList sums) ]
tests/Operations/NullableHashing.hs view
@@ -1,14 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- |-Regression tests for null-aware row hashing and row extraction.--`Nothing` in a nullable unboxed column must be a distinct value from any-`Just x` (so @distinct@/@groupBy@/joins do not merge them), and `toRowList`-must round-trip nulls faithfully. These pin the bug surfaced by the-category-theory set-algebra laws, where @Nothing@ was stored as an-uninitialised int and collided with @Just 0@.--}+-- | Regression tests for null-aware row hashing and row extraction. module Operations.NullableHashing where import qualified DataFrame as D@@ -18,7 +10,7 @@ import Test.HUnit maybeIntCol :: [Maybe Int] -> D.DataFrame-maybeIntCol xs = D.fromNamedColumns [("k", DI.fromList xs)]+maybeIntCol xs = D.fromColumns [("k", DI.fromList xs)] -- | distinct must keep @Nothing@ and @Just 0@ as two distinct rows. distinctSeparatesNullFromZero :: Test@@ -40,7 +32,7 @@ ( fst ( D.dimensions ( D.distinct- ( D.fromNamedColumns+ ( D.fromColumns [ ("k", DI.fromList [Just (0 :: Int), Nothing]) , ("v", DI.fromList ['a', 'a']) ]@@ -81,17 +73,13 @@ rebuilt = D.fromRows ["k"] (map (map snd) (toRowList df)) in TestCase (assertEqual "toRowList round-trip preserves nulls" df rebuilt) -{- | Hash robustness: @distinct@ must preserve every row of a dense grid of-small integers across several columns. A weak per-step hash collides badly on-adjacent integers and merges distinct rows; grouping trusts hash equality, so-this guards that the hash spreads such keys.--}+-- | distinct must preserve every row of a dense grid of small integers. denseIntGridNoCollisions :: Test denseIntGridNoCollisions = let d = [-4 .. 4 :: Int] rows = [(a, b, c) | a <- d, b <- d, c <- d] df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList (map (\(a, _, _) -> a) rows)) , ("b", DI.fromList (map (\(_, b, _) -> b) rows)) , ("c", DI.fromList (map (\(_, _, c) -> c) rows))@@ -103,16 +91,13 @@ (fst (D.dimensions (D.distinct df))) ) -{- | Join-hash robustness: a self inner-join on a dense grid of unique integer-keys must return exactly one match per row. Joins match purely by hash, so a-weak hash that collides distinct keys would emit spurious cross-matches.--}+-- | A self inner-join on a dense unique-key grid returns one match per row. joinDenseGridNoCollisions :: Test joinDenseGridNoCollisions = let d = [-4 .. 4 :: Int] rows = [(a, b) | a <- d, b <- d] df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList (map fst rows)) , ("b", DI.fromList (map snd rows)) ]@@ -129,7 +114,7 @@ let d = Nothing : map Just [-3 .. 3 :: Int] rows = [(a, b) | a <- d, b <- d] df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList (map fst rows)) , ("b", DI.fromList (map snd rows)) ]
tests/Operations/ParallelGroupBy.hs view
@@ -1,23 +1,22 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | The parallel-grouping correctness gate. Every case asserts that the-parallel partitioned grouping ('groupByPar') produces a 'Grouped' value-bit-for-bit identical to the sequential reference ('groupBySeq') —-'valueIndices', 'offsets' and 'rowToGroup' all equal — and that aggregating-through both yields the same result. Covered: single/multi key, Int/Text/Double-keys, heavy hash collisions, null patterns, and row counts straddling the-parallel threshold.--}+-- | Parallel grouping must be bit-for-bit identical to the sequential one. module Operations.ParallelGroupBy where +import qualified Data.ByteString as BS import qualified Data.Map.Strict as M import qualified Data.Text as T+import qualified Data.Text.Array as A+import qualified Data.Text.Encoding as TE+import Data.Text.Internal (Text (Text)) import qualified Data.Vector.Unboxed as VU import qualified DataFrame as D+import DataFrame.Expression.Operators ((./=.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI+import DataFrame.Internal.Data.PackedText (mkPackedContiguous) import qualified DataFrame.Internal.DataFrame as DD import DataFrame.Internal.Grouping (groupByPar, groupBySeq) @@ -46,7 +45,7 @@ -- | Build a frame of @n@ rows with several key columns and a value column. caseFrame :: Int -> DD.DataFrame caseFrame n =- D.fromNamedColumns+ D.fromColumns [ mkInt "ki" [i `mod` 997 | i <- [0 .. n - 1]] , mkText "kt" [T.pack ("g" ++ show (i `mod` 503)) | i <- [0 .. n - 1]] , mkDouble "kd" [fromIntegral (i `mod` 311) / 7 | i <- [0 .. n - 1]]@@ -60,7 +59,7 @@ -- | Collision-heavy small grid (forces hash-collision key re-verification). collisionFrame :: DD.DataFrame collisionFrame =- D.fromNamedColumns+ D.fromColumns [ mkInt "k1" (map (`mod` 5) [0 .. 999]) , mkInt "k2" (map (\i -> (i * 7) `mod` 3) [0 .. 999]) , mkInt "v" [0 .. 999]@@ -77,9 +76,7 @@ , ["ki", "kt", "kd", "kn"] ] -{- | Sizes straddling the 200k parallel threshold (the parallel path only kicks-in for the larger ones at runtime, but 'groupByPar' is exercised on all).--}+-- | Sizes straddling the 200k parallel threshold. sizes :: [Int] sizes = [1, 50, 4999, 200001, 350000] @@ -121,15 +118,10 @@ (groupByPar ["k1", "k2"] collisionFrame) ) -{- | The Q9 regression family: six derived/sum aggregations that together form-the moment sufficient statistics (count, Sx, Sy, Sxx, Syy, Sxy). The fused-'momentScatter' path must produce the same numbers as a deterministic oracle-computed from the raw data, at both -N1 and -N8 (parallel==sequential). The-input has columns the derive step turns into v1*v1, v1*v2, v2*v2.--}+-- | Frame for the moment-statistics aggregations checked against an oracle. momentFrame :: Int -> DD.DataFrame momentFrame n =- D.fromNamedColumns+ D.fromColumns [ mkInt "id2" [i `mod` 17 | i <- [0 .. n - 1]] , mkInt "id4" [(i * 3) `mod` 11 | i <- [0 .. n - 1]] , mkInt "v1" [(i * 7) `mod` 100 | i <- [0 .. n - 1]]@@ -165,11 +157,7 @@ (momentResult groupBySeq n) (momentResult groupByPar n) -{- | Independent oracle: recompute the six moment sums per group with plain-@Data.Map@ folds in original-row order and check the aggregated frame matches.-This pins the fused kernel against a non-scatter reference (counts exact, sums-fold in the same row order so they are byte-identical).--}+-- | Oracle recomputing the six moment sums per group with plain Map folds. momentOracle :: Int -> Test momentOracle n = TestCase $ do@@ -183,20 +171,12 @@ Nothing -> -1 assertEqual ("oracle group-row count n=" ++ show n) refN outN -{- | The low-cardinality DIRECT-INDEXED grouping fast path-('DataFrame.Internal.Grouping.Direct') fires from 'D.groupBy' on a single clean-small-range Int key. It emits groups in ascending key-value order rather than the-hash path's order, so we cannot compare index structures directly; instead we-assert it produces the SAME per-key aggregate values as the hash 'groupBySeq'-reference. The per-group v-sum, count, min and max are exact integers, so they-must match the hash path's results key-for-key (independent of group order).--}+-- | Direct-indexed grouping must give the same per-key aggregates as hashing. directGroupAggMatches :: Int -> Test directGroupAggMatches n = TestCase $ let df = caseFrame n v = F.col @Int "v"- ki = F.col @Int "ki" aggs = [ "vsum" F..= F.sum v , "vcount" F..= F.count v@@ -211,9 +191,7 @@ (keyedAgg hashRes) (keyedAgg directRes) -{- | Read an aggregated frame keyed on @ki@ into a 'M.Map' from key to the four-aggregate values, so two frames can be compared independent of group order.--}+-- | Read an aggregated frame into a Map keyed by group, ignoring group order. keyedAgg :: DD.DataFrame -> M.Map Int (Int, Int, Int, Int) keyedAgg res = M.fromList@@ -225,12 +203,39 @@ zip4 (a : as) (b : bs) (c : cs) (d : ds) = (a, b, c, d) : zip4 as bs cs ds zip4 _ _ _ _ = [] +-- | A packed UTF-8 column, the layout the Parquet reader produces.+packedCol :: [T.Text] -> DI.Column+packedCol ts =+ let bytes = BS.concat (map TE.encodeUtf8 ts)+ offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts)+ Text arr _ _ = TE.decodeUtf8 bytes+ in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs))++-- | Skewed packed-text keys with a few hot keys, grouped in parallel.+skewedTextParity :: Bool -> Test+skewedTextParity filtered = TestCase $ do+ let n = 300000+ key i+ | i `mod` 3 == 0 = ""+ | i `mod` 5 == 0 = "hot"+ | i `mod` 7 == 0 = "warm"+ | otherwise = T.pack ("k" ++ show ((i * 7919) `mod` 40000))+ df0 =+ D.fromColumns [("k", packedCol (map key [0 .. n - 1])), mkInt "v" [0 .. n - 1]]+ df = if filtered then D.filterWhere (F.col @T.Text "k" ./=. "") df0 else df0+ assertBool+ "skewed packed text: par == seq"+ (sameGrouping (groupByPar ["k"] df) (groupBySeq ["k"] df))+ tests :: [Test] tests =- [ TestLabel ("parity " ++ show n ++ " " ++ show keys) (parityFor n keys)- | n <- sizes- , keys <- keySets+ [ TestLabel ("skewedTextParity " ++ show f) (skewedTextParity f)+ | f <- [False, True] ]+ ++ [ TestLabel ("parity " ++ show n ++ " " ++ show keys) (parityFor n keys)+ | n <- sizes+ , keys <- keySets+ ] ++ [TestLabel ("aggParity " ++ show n) (aggParityFor n) | n <- [50, 200001]] ++ [TestLabel "collisionParity" collisionParity] ++ [TestLabel ("momentParity " ++ show n) (momentParity n) | n <- [3, 50, 200001]]
tests/Operations/ParallelJoin.hs view
@@ -1,13 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- | The parallel-join correctness gate. Each case asserts that the parallel-chunked-probe kernels ('parInnerKernel' \/ 'parLeftKernel') produce index-vectors /bit-for-bit identical/ to the sequential reference kernels, and that-the assembled join through the public 'innerJoin' \/ 'leftJoin' matches a-sequentially-forced reference. Covered: int\/text keys, many-to-many duplicate-keys, unmatched left rows, null patterns, and row counts straddling the-parallel threshold.--}+-- | Parallel join kernels must match the sequential reference bit for bit. module Operations.ParallelJoin where import qualified Data.Text as T@@ -31,9 +24,7 @@ mkHashes :: [Int] -> VU.Vector Int mkHashes = VU.fromList -{- | Synthetic hash vectors. Many collisions (mod), duplicates on both sides,-and sizes that exceed the parallel threshold so the parallel path engages.--}+-- | Synthetic hash vectors with collisions, duplicates and large sizes. probeBuildPairs :: [(String, HashPair)] probeBuildPairs = [ ("tiny", (mkHashes [1, 2, 3], mkHashes [2, 3, 4]))@@ -52,10 +43,7 @@ , mkHashes [i * 3 | i <- [0 .. 120000 :: Int]] ) )- , -- Small-build (cache-resident) build side probed by a very large probe:- -- the parallel small-build-probe lever (medium-factor regime). The probe- -- exceeds parProbeThreshold so the parallel path engages with a tiny- -- read-only shared index. Parity here is the new correctness gate.+ , -- Small-build side probed by a very large probe; parity is the gate. ( "small-build-big-probe" ,@@ -86,22 +74,19 @@ -- | Build a frame large enough to cross the parallel threshold. ordersFrame :: Int -> D.DataFrame ordersFrame n =- D.fromNamedColumns+ D.fromColumns [ ("cid", D.fromList [i `mod` 7000 | i <- [0 .. n - 1]]) , ("amount", D.fromList [fromIntegral i * 1.5 :: Double | i <- [0 .. n - 1]]) ] custFrame :: Int -> D.DataFrame custFrame n =- D.fromNamedColumns+ D.fromColumns [ ("cid", D.fromList [0 .. n - 1]) , ("region", D.fromList [T.pack ('r' : show (i `mod` 5)) | i <- [0 .. n - 1]]) ] -{- | End-to-end: the assembled inner/left join over a frame that exceeds the-parallel threshold equals the same join sorted (order-independent value check),-and the result row count is stable.--}+-- | End to end: a large join equals the sorted join and has a stable row count. endToEndInner :: Test endToEndInner = TestCase $@@ -123,30 +108,22 @@ 600000 (D.nRows joined) -{- | A probe frame whose join key is TEXT and whose row count crosses the-parallel row-hash threshold, so the inner join hashes the key in parallel.-Exercises the parallel text/factor-key hash path (the medium-factor lever) end-to end: a factor key with @k@ distinct values mapped over @n@ rows.--}+-- | Probe frame with a text join key and enough rows for parallel hashing. factorOrdersFrame :: Int -> Int -> D.DataFrame factorOrdersFrame n k =- D.fromNamedColumns+ D.fromColumns [ ("fk", D.fromList [T.pack ('f' : show (i `mod` k)) | i <- [0 .. n - 1]]) , ("amount", D.fromList [fromIntegral i * 2.0 :: Double | i <- [0 .. n - 1]]) ] factorDimFrame :: Int -> D.DataFrame factorDimFrame k =- D.fromNamedColumns+ D.fromColumns [ ("fk", D.fromList [T.pack ('f' : show i) | i <- [0 .. k - 1]]) , ("label", D.fromList [T.pack ('L' : show i) | i <- [0 .. k - 1]]) ] -{- | Inner join on a TEXT key over a frame past the parallel-hash threshold: the-every-key-matches case must keep all @n@ probe rows, confirming the parallel-text hashing buckets identically to a sequential pass (a miscomputed parallel-hash would drop matches and shrink the row count).--}+-- | Inner join on a text key keeps all n probe rows under parallel hashing. endToEndFactorInner :: Test endToEndFactorInner = TestCase $@@ -159,13 +136,7 @@ n (D.nRows joined) -{- | Small-build (cache-resident dim), very large probe (> parProbeThreshold) on-a TEXT/factor key, the medium-factor lever. The public 'innerJoin' takes the-parallel small-build-probe path here; every probe key matches a dim key, so a-mis-partitioned parallel probe that dropped or duplicated matches would change-the kept row count. The bit-identity of the parallel and sequential kernels is-pinned separately by 'innerKernelParity' on the @small-build-big-probe@ shape.--}+-- | Small-build, very large probe inner join on a text key keeps every match. endToEndSmallBuildFactorInner :: Test endToEndSmallBuildFactorInner = TestCase $@@ -179,11 +150,7 @@ n (D.nRows joined) -{- | Small-build, large-probe LEFT join on the factor key: every probe row is-kept (matched or sentinel). Exercises the parallel small-build 'parLeftKernel'-through the public 'leftJoin'. With @k@ distinct dim keys and probe keys in-@[0, 2k)@, roughly half the probe rows miss and carry a Nothing.--}+-- | Small-build, large-probe left join keeps every probe row, matched or not. endToEndSmallBuildFactorLeft :: Test endToEndSmallBuildFactorLeft = TestCase $
tests/Operations/Provenance.hs view
@@ -15,7 +15,7 @@ -- Base frame with no derived columns. base :: D.DataFrame base =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList [1 .. 5 :: Int]) , ("y", DI.fromList [2 .. 6 :: Int]) ]@@ -24,8 +24,6 @@ withZ :: D.DataFrame withZ = D.derive "z" (F.col @Int "x" + F.col "y") base --- ── insertColumn ──────────────────────────────────────────────────────────────- -- Inserting a new column must not wipe provenance of existing derived columns. insertPreservesProvenance :: Test insertPreservesProvenance =@@ -53,8 +51,6 @@ "sibling column w expression should be preserved" (M.member "w" (DI.derivingExpressions df3)) --- ── derive ────────────────────────────────────────────────────────────────────- -- derive adds the expression to derivingExpressions. deriveTracksExpression :: Test deriveTracksExpression =@@ -86,8 +82,6 @@ (M.size (DI.derivingExpressions df)) ) --- ── deriveWithExpr ────────────────────────────────────────────────────────────- -- deriveWithExpr should also track the expression. deriveWithExprTracksExpression :: Test deriveWithExprTracksExpression =@@ -98,8 +92,6 @@ (M.member "z" (DI.derivingExpressions df)) ) --- ── showDerivedExpressions ────────────────────────────────────────────────────- -- A frame with no derived columns returns identity provenance for each column. showDerivedEmpty :: Test showDerivedEmpty =@@ -119,8 +111,6 @@ ("z" `elem` map fst (D.showDerivedExpressions withZ)) ) --- ── Semigroup (<>) provenance propagation ─────────────────────────────────────- -- Vertical merge preserves expressions from both sides. semiGroupPreservesLeft :: Test semiGroupPreservesLeft =@@ -172,12 +162,10 @@ (M.member "z" (DI.derivingExpressions (D.empty <> withZ))) ) --- ── Horizontal merge (|||) provenance propagation ────────────────────────────- horizontalMergePreservesLeft :: Test horizontalMergePreservesLeft = let dfW = D.derive "w" (F.col @Int "y") base- extra = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])]+ extra = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] merged = dfW ||| extra in TestCase ( assertBool@@ -187,12 +175,12 @@ horizontalMergePreservesRight :: Test horizontalMergePreservesRight =- let extra = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])]+ let extra = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] dfW = D.derive "w" (F.col @Int "y")- (D.fromNamedColumns [("y", DI.fromList [2 .. 6 :: Int])])+ (D.fromColumns [("y", DI.fromList [2 .. 6 :: Int])]) merged = extra ||| dfW in TestCase ( assertBool@@ -203,7 +191,7 @@ horizontalMergePreservesBoth :: Test horizontalMergePreservesBoth = let dfZ = withZ- dfW = D.fromNamedColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])]+ dfW = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])] -- give dfW a derived column on a separate base dfWD = D.derive "w" (F.col @Int "q") dfW merged = dfZ ||| dfWD
tests/Operations/ReadCsv.hs view
@@ -91,9 +91,7 @@ assertFailure "expected UnboxedColumn for 'girls' via fallback InferFromSample 10" --- File: a,b,c header; row "1,2,3,EXTRA"--- Total delimiters: 3 + 4 = 7; 7 div 3 = 2, numRow=1--- Row-1 strides 3,4,5 → "1","2","3" — EXTRA (stride 6) is never accessed+-- Row "1,2,3,EXTRA": the extra cell (stride 6) is never accessed. testExtraFields :: Test testExtraFields = TestLabel "malformed_extra_fields" $ TestCase $ do df <- readCsvNoInfer ("./tests/data/unstable_csv/" <> "extra_fields.csv")@@ -107,9 +105,7 @@ (D.fromList @T.Text ["3"]) col --- Inference + EitherRead: first rows of "score" are clean ints, later rows are--- "abc" / empty / "N/A". The sample is limited to the clean prefix so the--- assumption is Int; non-parsing later rows become Left carrying raw text.+-- Inference + EitherRead: later non-parsing rows become Left with raw text. eitherReadCsvInference :: Test eitherReadCsvInference = TestLabel "csv_eitherRead_inference" $ TestCase $ do df <-@@ -149,8 +145,7 @@ expectedId col Nothing -> assertFailure "id column missing"- -- 'name' has a row containing "N/A"; under EitherRead the text is just- -- a value so it lands in Right. Empty cells would become Left "".+ -- Under EitherRead 'name' text is just a value, so it lands in Right. let expectedName :: Column expectedName = D.fromList @(Either T.Text T.Text)@@ -170,9 +165,7 @@ D.readCsvWithOpts D.defaultReadOptions{D.safeRead = D.MaybeRead} "./tests/data/unstable_csv/either_read_mixed.csv"- -- "score" is not cleanly parseable as Int because "abc" fails; however the- -- BS inference walks all rows and falls back to Text when needed. The- -- relevant invariant for MaybeRead is that the column is nullable.+ -- The invariant for MaybeRead is that the column is nullable. case getColumn "score" df of Just (UnboxedColumn (Just _) _) -> pure () -- Int with bitmap Just (BoxedColumn (Just _) _) -> pure () -- Text-backed with bitmap@@ -184,8 +177,7 @@ <> " with no bitmap" Nothing -> assertFailure "score column missing" --- NoSafeRead leaves columns un-bitmapped when every row parses (id column is--- all ints) but falls back to a Text column for mixed rows like "score".+-- NoSafeRead: all-int columns stay un-bitmapped; mixed rows fall back to Text. noSafeReadCsv :: Test noSafeReadCsv = TestLabel "csv_noSafeRead" $ TestCase $ do df <-@@ -200,9 +192,7 @@ <> columnTypeString col Nothing -> assertFailure "id column missing" --- Lazy CSV reader + EitherRead: goes through a different pipeline--- (MutableColumn + null indices). Every row of 'id' parses cleanly, so the--- column is @Either Text Int@ with every value in 'Right'.+-- Lazy CSV reader + EitherRead: 'id' is @Either Text Int@ with all Right. lazyEitherReadCsv :: Test lazyEitherReadCsv = TestLabel "lazy_csv_eitherRead" $ TestCase $ do (df, _) <-@@ -221,9 +211,7 @@ col Nothing -> assertFailure "id column missing (lazy reader)" --- Per-column overrides (eager): 'id' is strict Int (NoSafeRead), 'score' is--- Either Text Int (EitherRead) recording the raw bytes for failures, 'name'--- falls back to the default (MaybeRead) and becomes a nullable Text column.+-- Per-column overrides (eager): 'id' strict Int, 'score' Either, 'name' Maybe. perColumnEagerOverrides :: Test perColumnEagerOverrides = TestLabel "csv_perColumnOverrides_eager" $ TestCase $ do df <-@@ -265,9 +253,7 @@ expectedScore col Nothing -> assertFailure "score column missing"- -- 'name': default MaybeRead kicks in → nullable column (bitmap attached).- -- Every cell is non-empty so the bitmap is all-valid, but the optional- -- wrap is still applied.+ -- 'name': default MaybeRead gives a nullable column with a bitmap. case getColumn "name" df of Just (BoxedColumn (Just _) _) -> pure () Just (PackedText (Just _) _) -> pure ()@@ -277,8 +263,7 @@ <> columnTypeString col Nothing -> assertFailure "name column missing" --- Per-column overrides (lazy): same configuration as the eager test, but--- exercises the lazy freezer's resolution of per-column modes.+-- Per-column overrides (lazy): same configuration as the eager test. perColumnLazyOverrides :: Test perColumnLazyOverrides = TestLabel "csv_perColumnOverrides_lazy" $ TestCase $ do (df, _) <-@@ -300,9 +285,7 @@ "lazy NoSafeRead override for 'id' → bare UnboxedColumn, got " <> columnTypeString col Nothing -> assertFailure "id column missing (lazy)"- -- EitherRead on 'score': every row is Right (first row was '10' → Int- -- builder), though rows that failed parsing are captured as Left via the- -- nulls list. We only assert the column is Boxed-of-Either.+ -- EitherRead on 'score': we only assert the column is Boxed-of-Either. case getColumn "score" df of Just (BoxedColumn Nothing _) -> pure () Just col ->@@ -311,8 +294,7 @@ <> columnTypeString col Nothing -> assertFailure "score column missing (lazy)" --- Default=NoSafeRead, override single column 'score' → MaybeRead.--- 'id' and 'name' keep the NoSafeRead default; 'score' becomes nullable.+-- Default NoSafeRead with 'score' overridden to MaybeRead (nullable). overrideNoSafeReadDefaultWithMaybeRead :: Test overrideNoSafeReadDefaultWithMaybeRead = TestLabel "csv_override_noSafeRead_default_MaybeRead" $ TestCase $ do@@ -334,17 +316,14 @@ -- 'score': MaybeRead override → nullable (bitmap present). case getColumn "score" df of Just (UnboxedColumn (Just _) _) -> pure () -- Int with bitmap- Just (BoxedColumn (Just _) _) -> pure () -- fallback Text with bitmap+ Just (BoxedColumn (Just _) _) -> pure () Just (PackedText (Just _) _) -> pure () -- packed Text with bitmap Just col -> assertFailure $ "'score' MaybeRead override should yield nullable column, got " <> columnTypeString col Nothing -> assertFailure "score column missing"- -- 'name': NoSafeRead default → Text; may have bitmap when data has- -- null cells (row 8 has "N/A" which is in missingIndicators).- -- ensureOptional is NOT forced, but a bitmap is still created when- -- the builder detects actual nulls.+ -- 'name': NoSafeRead default; a bitmap appears only if nulls exist. case getColumn "name" df of Just (BoxedColumn _ _) -> pure () Just (PackedText _ _) -> pure ()@@ -519,8 +498,7 @@ "lazy 'id' NoSafeRead → bare UnboxedColumn, got " <> columnTypeString col Nothing -> assertFailure "id missing (lazy)"- -- 'score': EitherRead override → BoxedColumn of Either values.- -- Lazy reader infers Int from first row; rows that fail become Left.+ -- 'score': EitherRead override gives BoxedColumn of Either values. case getColumn "score" df of Just (BoxedColumn Nothing _) -> pure () Just col ->
tests/Operations/Record.hs view
@@ -49,7 +49,7 @@ "./tests/data/alltypes_plain.parquet" ) --- Nullable fields (Maybe Text -> RNullableBoxed; Maybe Int -> RNullableUnboxed).+-- Nullable (Maybe Text -> RNullableBoxed, Maybe Int -> RNullableUnboxed). data User = User { userId :: Int64 , userName :: Maybe T.Text@@ -112,8 +112,8 @@ instance DT.HasSchema Foo where type Schema Foo = FooSchema- toColumns = DT.genericToColumns- fromColumns = DT.genericFromColumns+ recordsToColumns = DT.genericToColumns+ recordsFromFrame = DT.genericFromColumns orderSample :: [Order] orderSample =@@ -182,7 +182,7 @@ typeMismatchError :: Test typeMismatchError = TestCase $ do let badDf =- D.fromNamedColumns+ D.fromColumns [ ("order_id", DI.fromList ([1, 2, 3] :: [Int])) , ("region", DI.fromList (["us", "eu", "ap"] :: [T.Text])) , ("amount", DI.fromList ([10.0, 20.5, 30.0] :: [Double]))@@ -197,7 +197,7 @@ missingColumnError :: Test missingColumnError = TestCase $ do let badDf =- D.fromNamedColumns+ D.fromColumns [ ("region", DI.fromList (["us"] :: [T.Text])) , ("amount", DI.fromList ([10.0] :: [Double])) ]
+ tests/Operations/RowWise.hs view
@@ -0,0 +1,148 @@+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TypeApplications #-}++-- | An expression means what it computes for one row at a time.+module Operations.RowWise (tests) where++import Control.Exception (throw)+import qualified Data.Text as T+import qualified Data.Vector as V++import qualified DataFrame as D+import DataFrame.Expression.Operators ((./=.), (.<=.), (.>.))+import qualified DataFrame.Functions as F+import DataFrame.Internal.Column (Columnable, TypedColumn (..), toVector)+import DataFrame.Internal.Expression (Expr (..))+import DataFrame.Internal.Interpreter (interpret)++import Test.HUnit++-- | The second size is above the 200k threshold where loops run in parallel.+sizes :: [Int]+sizes = [60, 200003]++xs, ys :: Int -> [Int]+xs n = take n (cycle [7, -3, 0, 12, 5, -8, 1])+ys n = take n (cycle [2, 0, -1, 0, 3])++ds :: Int -> [Double]+ds n = take n (cycle [0.5, -1.5, 2.25, 100, -0.0, 3])++names :: Int -> [T.Text]+names n = take n (cycle ["Eco", "Business", "", "Eco Plus"])++maybes :: Int -> [Maybe Int]+maybes n = take n (cycle [Just 1, Nothing, Just 3, Nothing, Just 5, Just 6])++frame :: Int -> D.DataFrame+frame n =+ D.fromColumns+ [ "x" D.=: xs n+ , "y" D.=: ys n+ , "d" D.=: ds n+ , "name" D.=: names n+ , "m" D.=: maybes n+ , "flag" D.=: map (fmap (> 2)) (maybes n)+ ]++eval :: forall a. (Columnable a) => Int -> Expr a -> Either String [a]+eval n e = case interpret @a (frame n) e of+ Left err -> Left (show err)+ Right (TColumn col) -> Right (V.toList (either throw id (toVector @a @V.Vector col)))++x, y :: Expr Int+x = F.col "x"+y = F.col "y"++d :: Expr Double+d = F.col "d"++-- | The expression's rows equal the reference rows, at every size.+rows :: (Columnable a) => String -> Expr a -> (Int -> [a]) -> [Test]+rows label e expected =+ [ TestCase+ (assertEqual (label ++ " n=" ++ show n) (Right (expected n)) (eval n e))+ | n <- sizes+ ]++tests :: [Test]+tests =+ concat+ [ rows+ "a guard protects its branch"+ (If (y ./=. F.lit 0) (F.div x y) (F.lit 0))+ (\n -> [if b /= 0 then a `div` b else 0 | (a, b) <- zip (xs n) (ys n)])+ , rows+ "a guarded branch may repeat a failing operation"+ (If (y ./=. F.lit 0) (F.div x y + F.div x y) (F.lit 0))+ (\n -> [if b /= 0 then 2 * (a `div` b) else 0 | (a, b) <- zip (xs n) (ys n)])+ , rows+ "a null in the branch not chosen is ignored"+ (If (x .>. F.lit 0) (F.col @(Maybe Int) "m") (F.lit (Just 0)))+ (\n -> [if a > 0 then m else Just 0 | (a, m) <- zip (xs n) (maybes n)])+ , rows+ "nested conditions"+ ( If+ (x .>. F.lit 0)+ (If (y .>. F.lit 0) (x + y) (x - y))+ (If (y .>. F.lit 1) (x * y) x)+ )+ ( \n ->+ [ if a > 0 then (if b > 0 then a + b else a - b) else (if b > 1 then a * b else a)+ | (a, b) <- zip (xs n) (ys n)+ ]+ )+ , rows+ "an aggregate inside a branch sees every row"+ (If (d .<=. F.lit 1) (d - F.mean d) (F.lit 0))+ ( \n ->+ let mean = sum (ds n) / fromIntegral n+ in [if v <= 1 then v - mean else 0 | v <- ds n]+ )+ , rows+ "text branches"+ (If (x .>. F.lit 0) (F.lift T.toUpper (F.col @T.Text "name")) (F.lit "none"))+ (\n -> [if a > 0 then T.toUpper t else "none" | (a, t) <- zip (xs n) (names n)])+ , rows+ "nullable branches"+ ( If+ (x .>. F.lit 0)+ (F.lift (fmap (+ 1)) (F.col @(Maybe Int) "m"))+ (F.col @(Maybe Int) "m")+ )+ (\n -> [if a > 0 then fmap (+ 1) m else m | (a, m) <- zip (xs n) (maybes n)])+ , rows+ "a repeated subexpression"+ ((x + y) * (x + y) - (x + y))+ (\n -> [(a + b) * (a + b) - (a + b) | (a, b) <- zip (xs n) (ys n)])+ , rows+ "a subexpression repeated across branches"+ (If (y .>. F.lit 0) ((x + y) * F.lit 2) ((x + y) * F.lit 3))+ (\n -> [if b > 0 then (a + b) * 2 else (a + b) * 3 | (a, b) <- zip (xs n) (ys n)])+ , rows+ "different functions under one name are not merged"+ (F.lift (+ 1) x + F.lift (* 2) x)+ (\n -> [(a + 1) + (a * 2) | a <- xs n])+ ,+ [ TestCase+ ( assertBool+ "a missing column is reported even in a branch never chosen"+ ( either+ (const True)+ (const False)+ (eval 60 (If (F.lit True) x (F.col @Int "absent")))+ )+ )+ , TestCase+ ( assertBool+ "a null condition is an error"+ ( either+ (const True)+ (const False)+ (eval 60 (If (F.col @Bool "flag") x y))+ )+ )+ ]+ ]
tests/Operations/SetOps.hs view
@@ -9,29 +9,27 @@ import Test.HUnit -{- | Sort by the integer key column so set results (which come out in-hash-bucket order) can be compared deterministically.--}+-- | Sort by the integer key so hash-ordered set results compare stably. sortByA :: D.DataFrame -> D.DataFrame sortByA = D.sortBy [D.Asc (F.col @Int "A")] dfA :: D.DataFrame dfA =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList [1 :: Int, 2, 3, 3]) , ("B", DI.fromList ['a', 'b', 'c', 'c']) ] dfB :: D.DataFrame dfB =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList [3 :: Int, 4]) , ("B", DI.fromList ['c', 'd']) ] expect :: [Int] -> [Char] -> D.DataFrame expect as bs =- D.fromNamedColumns+ D.fromColumns [ ("A", DI.fromList as) , ("B", DI.fromList bs) ]
tests/Operations/Shuffle.hs view
@@ -15,7 +15,7 @@ testDataFrame :: D.DataFrame testDataFrame =- D.fromNamedColumns+ D.fromColumns [ ("numbers", D.fromList @Int [1 .. 26]) ] @@ -44,8 +44,7 @@ (D.columnNames testDataFrame) ) --- Test that un-shuffling restores the original dataframe--- which is known to be sorted in this case+-- Un-shuffling restores the original (sorted) dataframe. shufflePreservesData :: Test shufflePreservesData = let gen = mkStdGen 1234@@ -54,8 +53,7 @@ in TestCase (assertEqual "sort recovers initial numbers" testDataFrame sortedShuffled) --- Test that shuffling isn't doing anything sneaky with summoning--- random numbers somehow+-- Shuffling must not summon random numbers somehow. shuffleSameSeedIsSameShuffle :: Test shuffleSameSeedIsSameShuffle = let gen = mkStdGen 1234@@ -103,24 +101,13 @@ (shuffledIndices (mkStdGen 7) 1) ) -{- | Chi-squared statistic of observed counts against a flat expectation:-sum over cells of (observed - expected)^2 / expected.--}+-- | Chi-squared statistic of observed counts against a flat expectation. chiSquared :: [Int] -> Double chiSquared counts = let expected = fromIntegral (sum counts) / fromIntegral (length counts) in sum [(fromIntegral o - expected) ^ (2 :: Int) / expected | o <- counts] -{- | Every permutation of n items is equally likely under a uniform shuffle,-so the counts over all n! outcomes are chi-squared with n! - 1 degrees of-freedom. Testing the whole permutation, rather than one position at a time,-also catches a shuffle whose positions are individually uniform but-correlated. Seeds are fixed, so the sample -- and the verdict -- is-deterministic.--n = 5 gives 120 outcomes; 12000 draws puts 100 in each on average. The bound-is the 0.999 quantile of chi-squared with 119 degrees of freedom.--}+-- | Every permutation of n items is equally likely under a uniform shuffle. shufflePermutationsAreUniform :: Test shufflePermutationsAreUniform = let n = 5@@ -137,15 +124,7 @@ (stat < 172.4) ) -{- | The frequency test from Knuth 3.3.2: each item lands in each position with-probability 1/n, so the n x n position-by-item table is chi-squared with-(n - 1)^2 degrees of freedom. A larger n than the permutation test can afford,-to catch bias that only shows at scale, such as a shuffle that leaves a-suffix untouched or never leaves an item in place.--n = 10 and 5000 draws put 500 in each cell. The bound is the 0.999 quantile of-chi-squared with 81 degrees of freedom.--}+-- | Knuth's frequency test: each item lands in each position with chance 1/n. shufflePositionsAreUniform :: Test shufflePositionsAreUniform = let n = 10
tests/Operations/Sort.hs view
@@ -23,11 +23,11 @@ ] testData :: D.DataFrame-testData = D.fromNamedColumns values+testData = D.fromColumns values moreTestData :: D.DataFrame moreTestData =- D.fromNamedColumns+ D.fromColumns [ ("test1", DI.fromList $ replicate 10 (0 :: Int) ++ replicate 10 1) , ("test2", DI.fromList $ [1 :: Int .. 10] ++ [1 .. 10]) ]@@ -37,7 +37,7 @@ TestCase ( assertEqual "Sorting rows by ascending works as intended"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList [(1 :: Int) .. 26]) , ("test2", DI.fromList ['a' .. 'z']) ]@@ -50,7 +50,7 @@ TestCase ( assertEqual "Sorting rows by descending works as intended"- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList $ reverse [(1 :: Int) .. 26]) , ("test2", DI.fromList $ reverse ['a' .. 'z']) ]@@ -72,7 +72,7 @@ TestCase ( assertEqual "Sorting moreTestData by Desc of test2 reverses the order of the second column."- ( D.fromNamedColumns+ ( D.fromColumns [ ("test1", DI.fromList $ replicate 10 (0 :: Int) ++ replicate 10 1) , ("test2", DI.fromList $ [10 :: Int, 9 .. 1] ++ [10, 9 .. 1]) ]@@ -91,7 +91,7 @@ compoundTestData :: D.DataFrame compoundTestData =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList ([3, 1, 4, 1, 5] :: [Int])) , ("b", DI.fromList ([10, 40, 20, 30, 50] :: [Int])) ]@@ -101,7 +101,7 @@ TestCase ( assertEqual "Sorting by Asc (a + b) orders rows by the sum without leaking a synthetic column"- ( D.fromNamedColumns+ ( D.fromColumns [ ("a", DI.fromList ([3, 4, 1, 1, 5] :: [Int])) , ("b", DI.fromList ([10, 20, 30, 40, 50] :: [Int])) ]@@ -114,7 +114,7 @@ TestCase ( assertEqual "Sorting by Desc (b - a) orders rows by descending difference"- ( D.fromNamedColumns+ ( D.fromColumns [ ("a", DI.fromList ([5, 1, 1, 4, 3] :: [Int])) , ("b", DI.fromList ([50, 40, 30, 20, 10] :: [Int])) ]@@ -127,7 +127,7 @@ TestCase ( assertEqual "Mixing a compound Asc key with a bare Desc tie-breaker works"- ( D.fromNamedColumns+ ( D.fromColumns [ ("a", DI.fromList ([1, 1, 3, 4, 5] :: [Int])) , ("b", DI.fromList ([40, 30, 10, 20, 50] :: [Int])) ]
+ tests/Operations/SortMask.hs view
@@ -0,0 +1,102 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | The radix sort and parallel mask paths agree with simple references.+module Operations.SortMask where++import Data.Bits (shiftR, xor)+import Data.Function ((&))+import Data.Int (Int32)+import qualified Data.List as L+import Data.Ord (Down (..), comparing)+import qualified Data.Vector.Unboxed as VU++import qualified DataFrame as D+import qualified DataFrame.Functions as F+import DataFrame.Internal.Column (maskIndices)+import qualified DataFrame.Internal.Column as DI+import qualified DataFrame.Internal.DataFrame as DD+import Test.HUnit++-- | Deterministic pseudo-random values in @[0, range)@.+draws :: Int -> Int -> Int -> [Int]+draws seed range n = [mix (seed * 1000003 + i) `mod` range | i <- [1 .. n]]+ where+ mix z0 =+ let z1 = (z0 `xor` (z0 `shiftR` 30)) * (-4658895280553007687)+ z2 = (z1 `xor` (z1 `shiftR` 27)) * (-7723592293110705685)+ in abs (z2 `xor` (z2 `shiftR` 31))++-- | Row order of a stable sort, the order 'D.sortBy' must reproduce.+stableOrder :: (Ord k) => (a -> k) -> [a] -> [Int]+stableOrder key xs = map fst (L.sortBy (comparing (key . snd)) (zip [0 ..] xs))++sortedV :: D.DataFrame -> [Int]+sortedV df = maybe [] (DI.toList @Int) (DD.getColumn "v" df)++-- Many ties and negative keys, so stability and sign handling both show.+intSort :: Bool -> Test+intSort desc = TestCase $ do+ let n = 300000+ ks = map (subtract 500) (draws 7 1000 n)+ df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. n - 1 :: Int])]+ ord = if desc then D.Desc (F.col @Int "k") else D.Asc (F.col @Int "k")+ expected = if desc then stableOrder Down ks else stableOrder id ks+ assertEqual+ ("Int sort desc=" ++ show desc)+ expected+ (sortedV (df & D.sortBy [ord]))++narrowSort :: Test+narrowSort = TestCase $ do+ let ks = map (fromIntegral . subtract 50) (draws 8 100 5000) :: [Int32]+ df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. 4999 :: Int])]+ assertEqual+ "Int32 sort"+ (stableOrder id ks)+ (sortedV (df & D.sortBy [D.Asc (F.col @Int32 "k")]))++-- -0.0 and 0.0 compare equal, so they keep row order.+doubleSort :: Test+doubleSort = TestCase $ do+ let ks =+ [0.0, -0.0, 1.5, -2.25, 1 / 0, -(1 / 0), 0.0, 1.5, -1.0e-300, 1.0e-300] ::+ [Double]+ df =+ D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. length ks - 1])]+ assertEqual+ "Double sort"+ (stableOrder id ks)+ (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")]))+ assertEqual+ "Double sort desc"+ (stableOrder Down ks)+ (sortedV (df & D.sortBy [D.Desc (F.col @Double "k")]))++-- A NaN column keeps the comparison sort; this only checks it still sorts.+nanSort :: Test+nanSort = TestCase $ do+ let ks = [2, 0 / 0, 1, 3] :: [Double]+ df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. 3 :: Int])]+ assertEqual+ "NaN sort length"+ 4+ (length (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")])))++maskCases :: [Test]+maskCases =+ [ TestCase $+ let bs = map ((< t) . (`mod` 10)) (draws n 10 n)+ in assertEqual+ ("maskIndices n=" ++ show n ++ " t=" ++ show t)+ (VU.fromList [i | (i, True) <- zip [0 ..] bs])+ (maskIndices (VU.fromList bs))+ | n <- [0, 1, 1000, 300001]+ , t <- [0, 3, 10]+ ]++tests :: [Test]+tests =+ map+ (TestLabel "sortMask")+ ([intSort False, intSort True, narrowSort, doubleSort, nanSort] ++ maskCases)
tests/Operations/Statistics.hs view
@@ -191,7 +191,7 @@ 3 -- The three columns should be Statistics, A, and B ( D.nColumns ( D.summarize- ( D.fromNamedColumns+ ( D.fromColumns [ ("A", D.fromList [1 :: Int, 2]) , ("B", D.fromList [Just (1 :: Int), Nothing]) ]@@ -200,13 +200,10 @@ ) ) -{- | A sliced column's bitmap keeps whole bytes, so its trailing bits still-describe rows past the end of the slice. Counting non-null rows has to stop-at the column's length rather than fold the whole byte vector.--}+-- | A sliced nullable column must count non-null rows only up to its length. sixteenNullableRows :: D.DataFrame sixteenNullableRows =- D.fromNamedColumns+ D.fromColumns [ ("x", D.fromList (map (Just . fromIntegral) [1 .. 16 :: Int] :: [Maybe Double])) ] @@ -234,8 +231,7 @@ ) ) --- allMissing folds the same bitmap, so it needs the same length cutoff: the--- padding bits of a sliced all-null column would otherwise read as present.+-- allMissing folds the same bitmap, so it needs the same length cutoff. allMissingAfterSlice :: Test allMissingAfterSlice = TestCase@@ -247,14 +243,14 @@ allNullRows :: D.DataFrame allNullRows =- D.fromNamedColumns+ D.fromColumns [("x", DI.fromList (replicate 16 (Nothing :: Maybe Double)))] -- correlation correlationDf :: D.DataFrame correlationDf =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList [1 :: Int, 2, 3, 4, 5]) , ("y_pos", DI.fromList [1 :: Int, 2, 3, 4, 5]) , ("y_neg", DI.fromList [5 :: Int, 4, 3, 2, 1])
tests/Operations/Subset.hs view
@@ -60,8 +60,7 @@ fst (dataframeDimensions (D.range (a, b) df)) == expected where rows = fst (dataframeDimensions df)- -- Rows in [a, b) that actually exist. Clamps before subtracting so the- -- oracle itself cannot overflow on extreme endpoints.+ -- Rows in [a, b) that exist; clamps before subtracting to avoid overflow. lo = min (max a 0) rows hi = min (max b lo) rows expected = hi - lo@@ -96,7 +95,7 @@ prop_stratifiedSplit_deterministic :: DataFrame -> Bool prop_stratifiedSplit_deterministic _ = let df =- D.fromNamedColumns+ D.fromColumns [ ("label", Col.fromList (replicate 50 ("A" :: T.Text) ++ replicate 50 "B")) , ("val", Col.fromList ([1 .. 100] :: [Int])) ]@@ -105,7 +104,7 @@ strataDf :: DataFrame strataDf =- D.fromNamedColumns+ D.fromColumns [ ("label", Col.fromList (replicate 5 ("A" :: T.Text) ++ replicate 5 "B")) , ("val", Col.fromList ([1 .. 10] :: [Int])) ]@@ -132,7 +131,7 @@ unit_stratifiedSplit_singleRowStratum = TestCase $ let tinyDf =- D.fromNamedColumns+ D.fromColumns [ ("label", Col.fromList (["A", "A", "A", "A", "A", "B"] :: [T.Text])) , ("val", Col.fromList ([1 .. 6] :: [Int])) ]@@ -142,7 +141,7 @@ (fst $ dataframeDimensions tinyDf) (fst (dataframeDimensions tr) + fst (dataframeDimensions va)) --- | Count occurrences of a label in a column, expressed as a fraction of total rows.+-- | Count of a label in a column, as a fraction of total rows. labelProportion :: T.Text -> T.Text -> DataFrame -> Double labelProportion col label df = let total = fst (dataframeDimensions df)@@ -158,7 +157,7 @@ let aCount = 100 bCount = 50 df =- D.fromNamedColumns+ D.fromColumns [ ( "label" , Col.fromList (replicate aCount ("A" :: T.Text) ++ replicate bCount "B")@@ -191,7 +190,7 @@ (abs (vaProp - origProp) < tol) tenRows :: DataFrame-tenRows = fromNamedColumns [("x", Col.fromList ([0 .. 9] :: [Int]))]+tenRows = fromColumns [("x", Col.fromList ([0 .. 9] :: [Int]))] -- Endpoints that overflow Int if the length is computed before clamping. unit_rangeExtremeEndpoints :: Test
tests/Operations/Take.hs view
@@ -10,7 +10,7 @@ testData :: D.DataFrame testData =- D.fromNamedColumns+ D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) ]
+ tests/Operations/TextKernels.hs view
@@ -0,0 +1,81 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | contains and length on packed text agree with the Text functions per row.+module Operations.TextKernels (tests) where++import qualified Data.ByteString as BS+import qualified Data.Text as T+import qualified Data.Text.Array as A+import qualified Data.Text.Encoding as TE+import Data.Text.Internal (Text (Text))+import qualified Data.Vector.Unboxed as VU++import qualified DataFrame as D+import qualified DataFrame.Functions as F+import qualified DataFrame.Internal.Column as DI+import DataFrame.Internal.Data.PackedText (mkPackedContiguous)+import Test.HUnit++-- | Packed UTF-8 column of the given texts.+packed :: [T.Text] -> DI.Column+packed ts =+ let bytes = BS.concat (map TE.encodeUtf8 ts)+ offs = scanl (+) 0 (map (BS.length . TE.encodeUtf8) ts)+ Text arr _ _ = TE.decodeUtf8 bytes+ in DI.PackedText Nothing (mkPackedContiguous (arr :: A.Array) (VU.fromList offs))++rows :: [T.Text]+rows =+ [ "http://www.google.com/search?q=1"+ , ""+ , "goo"+ , "gle"+ , "Google Docs"+ , "ищу google здесь"+ , "ééé.google.é"+ , "a"+ , "g"+ , "gooogle"+ ]++frame :: D.DataFrame+frame = D.fromColumns [("t", packed rows)]++containsCase :: T.Text -> Test+containsCase needle =+ TestCase $+ assertEqual+ ("contains " ++ show needle)+ (map (T.isInfixOf needle) rows)+ ( D.columnAsList+ (F.col @Bool "m")+ (D.derive "m" (F.contains needle (F.col "t")) frame)+ )++lengthCase :: Test+lengthCase =+ TestCase $+ assertEqual+ "length"+ (map T.length rows)+ (D.columnAsList (F.col @Int "l") (D.derive "l" (F.length (F.col "t")) frame))++tests :: [Test]+tests =+ map (TestLabel "text kernels") $+ lengthCase+ : map+ containsCase+ [ "google"+ , "Google"+ , ".google."+ , ""+ , "g"+ , "gle"+ , "goo"+ , "здесь"+ , "é.g"+ , "x"+ , "gooogle!"+ ]
+ tests/Operations/TopBy.hs view
@@ -0,0 +1,51 @@+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | 'topBy' returns the window a full 'sortBy' would, for every key type.+module Operations.TopBy (tests) where++import qualified Data.Text as T++import qualified DataFrame as D+import qualified DataFrame.Functions as F+import Test.HUnit++frame :: D.DataFrame+frame =+ D.fromColumns+ [ "n" D.=: [(i * 7919) `mod` 97 | i <- [0 .. 999 :: Int]]+ , "d"+ D.=: [fromIntegral ((i * 31) `mod` 13) / 4 :: Double | i <- [0 .. 999 :: Int]]+ , "t" D.=: [T.pack (show ((i * 131) `mod` 53)) | i <- [0 .. 999 :: Int]]+ , "id" D.=: [0 .. 999 :: Int]+ ]++orders :: [(String, [D.SortOrder])]+orders =+ [ ("int asc", [D.Asc (F.col @Int "n")])+ , ("int desc", [D.Desc (F.col @Int "n")])+ , ("text asc", [D.Asc (F.col @T.Text "t")])+ ,+ ( "double desc, text asc"+ , [D.Desc (F.col @Double "d"), D.Asc (F.col @T.Text "t")]+ )+ , ("expr", [D.Asc (F.col @Int "n" * 3 - F.col @Int "id")])+ ]++windows :: [(Int, Int)]+windows =+ [(0, 1), (0, 10), (5, 10), (990, 10), (995, 20), (0, 1000), (0, 2000), (0, 0)]++windowCases :: [Test]+windowCases =+ [ TestCase $+ assertEqual+ (label ++ " " ++ show (off, n))+ (D.range (off, off + n) (D.sortBy order frame))+ (D.topBy off n order frame)+ | (label, order) <- orders+ , (off, n) <- windows+ ]++tests :: [Test]+tests = map (TestLabel "topBy") windowCases
tests/Operations/Typing.hs view
@@ -18,7 +18,7 @@ testData :: D.DataFrame testData =- D.fromNamedColumns+ D.fromColumns [ ("test1", DI.fromList ([1 .. 26] :: [Int])) , ("test2", DI.fromList ['a' .. 'z']) ]@@ -71,8 +71,7 @@ datesInput :: [T.Text] datesInput = T.pack . show <$> datesExpected --- PARSING TESTS-------- 1. SIMPLE CASES+-- PARSING TESTS: 1. SIMPLE CASES parseBools :: Test parseBools = let afterParse :: [Bool]@@ -409,7 +408,7 @@ parseDatesAndEmptyStringsWithoutSafeRead :: Test parseDatesAndEmptyStringsWithoutSafeRead =- -- Pattern: 3 dates, empty, 3 dates, empty, 3 dates, empty, 3 dates, empty, 3 dates, empty+ -- Pattern: 3 dates then an empty, repeated five times let groups = chunksOf3 datesExpected beforeParse = concatMap (\g -> map (T.pack . show) g ++ [""]) groups afterParse :: [Maybe Day]@@ -484,7 +483,7 @@ parseIntsAndDoublesAndNullishStringsWithoutSafeRead :: Test parseIntsAndDoublesAndNullishStringsWithoutSafeRead =- -- Nullish separators are literal text strings (not interpreted as null when safeRead=False)+ -- Nullish separators are literal text when safeRead=False let nullishSeps = ["Nothing", "N/A", "NULL", "null", "NAN"] intGrp = T.pack . show <$> ([1 .. 10] :: [Int]) dblGrps =@@ -675,7 +674,7 @@ parseIntsAndDoublesAndNullishStringsWithSafeRead :: Test parseIntsAndDoublesAndNullishStringsWithSafeRead =- -- Note: this test uses different special doubles and nullish separators than the safeRead=False version+ -- Uses different special doubles and nullish separators than safeRead=False let nullishSeps = ["Nothing", "N/A", "NULL", "null", "NAN"] intGrp = T.pack . show <$> ([1 .. 10] :: [Int]) dblGrps =@@ -736,7 +735,7 @@ parseIntsAndDoublesAndNullishAndEmptyStringsWithSafeRead :: Test parseIntsAndDoublesAndNullishAndEmptyStringsWithSafeRead =- -- Data: 4xN/A, "Nothing", "", then 10 ints (1-10), "", 10 ints (11-20), "", 10 ints (21-30), "", 3.14+ -- Data: 4xN/A, "Nothing", "", then ints 1-30 split by empties, 3.14 let beforeParse = ["N/A", "N/A", "N/A", "N/A", "Nothing", ""] ++ map (T.pack . show) ([1 .. 10] :: [Int])@@ -770,7 +769,7 @@ parseTextsAndEmptyAndNullishStringsWithSafeRead = TestCase $ do raw <- T.lines <$> TIO.readFile (typingDataDir <> "texts_with_empties_and_nullish.txt")- -- safeRead=True: empty strings and nullish tokens (NaN, Nothing, N/A) -> Nothing+ -- safeRead=True: empty strings and nullish tokens become Nothing let isNullish t = t `elem` ["NaN", "Nothing", "N/A", "nan", "null", "NULL", "NA", "na", "NAN"] afterParse = map (\t -> if t == "" || isNullish t then Nothing else Just t) raw expected = DI.fromVector $ V.fromList afterParse@@ -784,10 +783,6 @@ actual -- 3C. PARSING WITH SAFEREAD = EitherRead------ EitherRead wraps every column as @Either Text a@: successful parses become--- @Right v@; failures (including nullish/empty cells) become @Left <raw>@--- preserving the original input verbatim. parseBoolsWithEitherRead :: Test parseBoolsWithEitherRead =@@ -840,7 +835,7 @@ parseDoublesWithEitherRead :: Test parseDoublesWithEitherRead =- -- Sample is cleanly Double; "oops" / "" come after and must surface as Left.+ -- Sample is cleanly Double; later "oops" and "" must surface as Left. let beforeParse :: [T.Text] beforeParse = ["1.5", "2.0", "3.5", "4.25", "3.14", "oops", ""] afterParse :: [Either T.Text Double]@@ -908,12 +903,11 @@ actual ) --- parseDefaults honours per-column overrides: 'ages' stays strict Int,--- 'notes' is wrapped as Either Text Text.+-- parseDefaults honours per-column overrides of the default read mode. parseDefaultsPerColumnOverrides :: Test parseDefaultsPerColumnOverrides = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("ages", DI.fromList @T.Text ["1", "2", "3"]) , ("notes", DI.fromList @T.Text ["hello", "", "world"]) ]@@ -949,7 +943,7 @@ parseDefaultsNoSafeReadWithMaybeOverride :: Test parseDefaultsNoSafeReadWithMaybeOverride = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList @T.Text ["1", "2", "3"]) , ("y", DI.fromList @T.Text ["a", "", "c"]) ]@@ -979,7 +973,7 @@ parseDefaultsEitherReadWithNoSafeOverride :: Test parseDefaultsEitherReadWithNoSafeOverride = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("nums", DI.fromList @T.Text ["10", "20", "30"]) , ("tags", DI.fromList @T.Text ["foo", "", "baz"]) ]@@ -1010,7 +1004,7 @@ parseDefaultsEmptyOverrides :: Test parseDefaultsEmptyOverrides = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [("v", DI.fromList @T.Text ["1", "2", "3"])] opts1 = D.defaultParseOptions{D.sampleSize = 3} opts2 = opts1{D.parseSafeOverrides = []}@@ -1023,7 +1017,7 @@ parseDefaultsNonExistentOverride :: Test parseDefaultsNonExistentOverride = TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [("a", DI.fromList @T.Text ["1", "2", "3"])] opts = D.defaultParseOptions@@ -1042,8 +1036,7 @@ parseAllNullWithEitherRead :: Test parseAllNullWithEitherRead =- -- NoAssumption path: sample is all empty/nullish so inference can't- -- pick a numeric type. EitherRead still wraps as Either Text Text.+ -- NoAssumption path: EitherRead still wraps as Either Text Text. let beforeParse :: [T.Text] beforeParse = ["", "", "", "N/A", ""] afterParse :: [Either T.Text T.Text]@@ -1212,7 +1205,7 @@ parseIntsAndDoublesAndEmptyStringsAsDoublesWithOneExampleWithSafeReadOff :: Test parseIntsAndDoublesAndEmptyStringsAsDoublesWithOneExampleWithSafeReadOff =- -- Pattern: 10 empties, 50 ints (as int text), 50 doubles (as double text), 5 specials+ -- Pattern: 10 empties, 50 ints, 50 doubles, 5 specials let beforeParse = replicate 10 "" ++ intsInput@@ -1267,7 +1260,7 @@ parseIntsAndDoublesAndEmptyStringsAndNullishAsStringssWithOneExampleWithSafeReadOff :: Test parseIntsAndDoublesAndEmptyStringsAndNullishAsStringssWithOneExampleWithSafeReadOff =- -- Pattern: 10 empties, 5x(NaN,N/A), 20 ints (1-20), 30 doubles (1.0-30.0), 5 specials+ -- Pattern: 10 empties, 5x(NaN,N/A), 20 ints, 30 doubles, 5 specials let nullishPairs = concat (replicate 5 ["NaN", "N/A"]) beforeParse = replicate 10 ""
tests/Operations/VectorKernel.hs view
@@ -2,31 +2,10 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- | The vectorized scatter-accumulate kernel parity gate. Every case asserts-that 'D.aggregate' (which routes recognised reductions through the fast scatter-kernel) produces a result byte-identical to the same aggregation forced through-the expression interpreter ('interpretOnly'). Covered: sum/min/max/count over-Int and Double, mean, stddev/variance, top2Sum, median, the @max - min@-compound (Q7), and a mixed multi-aggregation over single and multi-key-groupings, on Int/Double/nullable/Text-key grids and sizes straddling the-parallel threshold.--Floating-point sums computed by the scatter follow the same left-to-right-group-order fold as the interpreter, so the two paths agree bit-for-bit here.-The PARALLEL path ('DataFrame.Internal.Aggregation.Kernel.Scatter') splits the work by-disjoint group-id range, and because each group's rows stay in their original-@valueIndices@ order within one worker's range, the per-group fold order is-unchanged from the sequential scatter — so the parallel path is also-byte-identical to the interpreter, asserted here at sizes above the 200k-parallel threshold (the test suite runs @-with-rtsopts=-N@, so those cases-exercise the multi-capability scatter).--} module Operations.VectorKernel where import Control.Exception (throw)-import qualified Data.List as L import qualified Data.Text as T-import qualified Data.Vector as V import qualified DataFrame as D import DataFrame.Errors (DataFrameException (..))@@ -34,9 +13,8 @@ import qualified DataFrame.Internal.Column as DI import DataFrame.Internal.DataFrame (GroupedDataFrame, insertColumn) import DataFrame.Internal.Expression (- AggStrategy (..), Expr (..),- UExpr (..),+ SomeExpr (..), ) import DataFrame.Internal.Interpreter ( AggregationResult (..),@@ -48,11 +26,11 @@ import Test.HUnit -- | Aggregate forcing every expression through the interpreter (no kernel).-interpretOnly :: [(T.Text, UExpr)] -> GroupedDataFrame -> D.DataFrame+interpretOnly :: [(T.Text, SomeExpr)] -> GroupedDataFrame -> D.DataFrame interpretOnly aggs gdf = let base = DA.aggregate [] gdf- f :: (T.Text, UExpr) -> D.DataFrame -> D.DataFrame- f (name, UExpr (expr :: Expr a)) d =+ f :: (T.Text, SomeExpr) -> D.DataFrame -> D.DataFrame+ f (name, SomeExpr (expr :: Expr a)) d = let value :: DI.Column value = case interpretAggregation @a gdf expr of Left e -> throw e@@ -61,20 +39,12 @@ in insertColumn name value d in foldl (flip f) base aggs --- | Per-group sum of the two largest values (the benchmark's Q8 reduction).-top2Sum :: Expr Double -> Expr Double-top2Sum = Agg (CollectAgg "top2Sum" f)- where- f :: V.Vector Double -> Double- f v = sum (take 2 (L.sortBy (flip compare) (V.toList v)))- grid :: Int -> D.DataFrame grid n =- D.fromNamedColumns+ D.fromColumns [ ("ki", DI.fromList [i `mod` 17 | i <- [0 .. n - 1]]) , ("kt", DI.fromList [T.pack ('g' : show (i `mod` 11)) | i <- [0 .. n - 1]])- , -- A high-cardinality key so the parallel group-range split sees many- -- small groups (alongside the low-cardinality 'ki'/'kt' few-big-groups).+ , -- A high-cardinality key so the group-range split sees many groups. ("kh", DI.fromList [(i * 2654435761) `mod` 50021 :: Int | i <- [0 .. n - 1]]) , ("vi", DI.fromList [(i * 31 + 7) `mod` 1000 - 500 :: Int | i <- [0 .. n - 1]]) , ("vj", DI.fromList [(i * 13) `mod` 97 :: Int | i <- [0 .. n - 1]])@@ -91,7 +61,7 @@ vd :: Expr Double vd = F.col @Double "vd" -aggSets :: [[(T.Text, UExpr)]]+aggSets :: [[(T.Text, SomeExpr)]] aggSets = [ ["s" F..= F.sum vi] , ["s" F..= F.sum vd, "c" F..= F.count vi]@@ -99,7 +69,6 @@ , ["mnd" F..= F.minimum vd, "mxd" F..= F.maximum vd] , ["m" F..= F.mean vi, "md" F..= F.mean vd] , ["sd" F..= F.stddev vd, "var" F..= F.variance vd]- , ["t2" F..= top2Sum vd] , ["med" F..= F.median vd] , ["diff" F..= (F.maximum vi - F.minimum vj)] ,@@ -115,7 +84,7 @@ keySets :: [[T.Text]] keySets = [["ki"], ["kt"], ["ki", "kt"], ["kh"]] -parityCase :: Int -> [T.Text] -> [(T.Text, UExpr)] -> Test+parityCase :: Int -> [T.Text] -> [(T.Text, SomeExpr)] -> Test parityCase n keys aggs = TestCase $ let df = grid n
tests/Operations/Window.hs view
@@ -13,10 +13,10 @@ import DataFrame.Expression.Operators import Test.HUnit --- Similar to example discussed in https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/+-- Similar to the example in sumsar.net's pandas-feels-clunky-from-r post. purchases :: D.DataFrame purchases =- D.fromNamedColumns+ D.fromColumns [ ( "country" , DI.fromList@@ -57,7 +57,7 @@ ) & D.sortBy [D.Asc (F.col @T.Text "country"), D.Asc (F.col @Double "amount")] expected =- D.fromNamedColumns+ D.fromColumns [ ( "country" , DI.fromList (["France", "France", "France", "UK", "UK", "US", "US"] :: [T.Text])@@ -99,7 +99,7 @@ ) where simpleData =- D.fromNamedColumns+ D.fromColumns [ ("group", DI.fromList (["A", "A", "B", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([10, 20, 30, 60, 90] :: [Double])) ]@@ -123,7 +123,7 @@ ) where simpleData =- D.fromNamedColumns+ D.fromColumns [ ("group", DI.fromList (["X", "X", "Y", "Y"] :: [T.Text])) , ("value", DI.fromList ([10, 20, 100, 200] :: [Int])) ]@@ -146,7 +146,7 @@ ) where simpleData =- D.fromNamedColumns+ D.fromColumns [ ("group", DI.fromList (["A", "A", "A", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([1, 2, 3, 4, 5] :: [Int])) ]@@ -169,7 +169,7 @@ ) where simpleData =- D.fromNamedColumns+ D.fromColumns [ ("group", DI.fromList (["A", "A", "B", "B"] :: [T.Text])) , ("value", DI.fromList ([10.0, 20.0, 100.0, 200.0] :: [Double])) ]@@ -184,7 +184,7 @@ Nothing -> error "deviation column not found" Just c -> DI.toList @Double c --- | Example from https://www.sumsar.net/blog/pandas-feels-clunky-when-coming-from-r/+-- | Example from the sumsar.net pandas-feels-clunky-from-r post. blogPostExampleGlobal :: Test blogPostExampleGlobal = TestCase@@ -202,10 +202,9 @@ & D.groupBy ["country"] & D.aggregate [F.sum (amount - discount) `as` "total"] & D.sortBy [D.Asc (F.col @T.Text "country")]- -- Global median = 60, threshold = 600 → removes 5000 and 800- -- France: (30-1)+(40-2)+(35-1) = 101, UK: (50-2)+(60-3) = 105, US: (100-5)+(200-10) = 285+ -- Global median 60 and threshold 600 remove 5000 and 800. expected =- D.fromNamedColumns+ D.fromColumns [ ("country", DI.fromList (["France", "UK", "US"] :: [T.Text])) , ("total", DI.fromList ([101, 105, 285] :: [Double])) ]@@ -228,7 +227,7 @@ & D.aggregate [F.sum (amount - discount) `as` "total"] & D.sortBy [D.Asc (F.col @T.Text "country")] expected =- D.fromNamedColumns+ D.fromColumns [ ("country", DI.fromList (["France", "UK", "US"] :: [T.Text])) , ("total", DI.fromList ([101, 105, 285] :: [Double])) ]
tests/Operations/WriteCsv.hs view
@@ -15,7 +15,7 @@ toCsvBasic :: Test toCsvBasic = TestLabel "toCsv_basic" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("name", DI.fromList @T.Text ["Alice", "Bob", "Charlie"]) , ("age", DI.fromList @Int [30, 25, 35]) ]@@ -25,7 +25,7 @@ toCsvWithNulls :: Test toCsvWithNulls = TestLabel "toCsv_withNulls" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ( "name" , DI.fromList @(Maybe T.Text) [Nothing, Just "Alice", Just "Bob", Just "Charlie"]@@ -46,7 +46,7 @@ toSeparatedTab :: Test toSeparatedTab = TestLabel "toSeparated_tab" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList @Int [1, 2]) , ("y", DI.fromList @Int [3, 4]) ]@@ -57,7 +57,7 @@ toCsvDouble :: Test toCsvDouble = TestLabel "toCsv_double" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("value", DI.fromList @Double [1.5, 2.0, 2.5]) ] expected = "value\n1.5\n2.0\n2.5\n"@@ -67,7 +67,7 @@ toCsvSingleColumn :: Test toCsvSingleColumn = TestLabel "toCsv_single_column" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("id", DI.fromList @Int [10, 20, 30]) ] expected = "id\n10\n20\n30\n"@@ -77,7 +77,7 @@ toCsvRoundTrip :: Test toCsvRoundTrip = TestLabel "toCsv_roundTrip" $ TestCase $ do let df =- D.fromNamedColumns+ D.fromColumns [ ("a", DI.fromList @Int [1, 2, 3]) , ("b", DI.fromList @T.Text ["hello", "world", "test"]) ]
tests/PackedTextMigration.hs view
@@ -1,13 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Regression tests for the PackedText migration: a string column stored as a-'DI.PackedText' (the form the CSV readers emit) must flow through the-decision-tree feature/target extraction and the full-outer-join key coalescing-without crashing, and must behave IDENTICALLY to the same data held as a boxed-@Text@ column. If a non-exhaustive @Column@ match regresses, the packed cases-either crash (test error) or diverge from the boxed baseline (assertion fails).--}+-- | Packed-text columns must behave identically to boxed Text columns. module PackedTextMigration (tests) where import Control.Monad (zipWithM_)@@ -29,9 +23,7 @@ import Test.HUnit -{- | Build a 'DI.PackedText' column directly from raw UTF-8 bytes + offsets,-exactly as the CSV readers do (mirrors @Internal.PackedText.packedFromTexts@).--}+-- | Build a packed-text column directly from UTF-8 bytes and offsets. packedFromTexts :: [T.Text] -> DI.Column packedFromTexts ts = let bytess = map (B.unpack . encodeUtf8) ts@@ -52,15 +44,11 @@ packed = packedFromTexts boxed = DI.fromList -{- | A decision tree using a TEXT FEATURE (exercises Cart.featuresOfColumn and the-categorical condition builders). The packed-feature tree must equal the-boxed-feature tree, proving the feature is used identically — not dropped or-crashed on.--}+-- | A decision tree on a packed text feature must equal the boxed-feature tree. treeOnTextFeature :: Test treeOnTextFeature = TestCase $ do let df mk =- D.fromNamedColumns+ D.fromColumns [ ("color", mk ["red", "red", "blue", "blue", "green", "green"]) , ("y", DI.fromList ([0, 0, 1, 1, 2, 2] :: [Double])) ]@@ -71,13 +59,11 @@ (D.prettyPrint (predict boxedTree)) (D.prettyPrint (predict packedTree)) -{- | A classifier whose TARGET is a packed-text column (exercises-Cart.cartTargetLabels). Packed target must give the same tree as a boxed one.--}+-- | A classifier with a packed-text target gives the same tree as a boxed one. classifierOnTextTarget :: Test classifierOnTextTarget = TestCase $ do let df mk =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList ([1, 2, 3, 4, 5, 6] :: [Double])) , ("label", mk ["a", "a", "a", "b", "b", "b"]) ]@@ -88,17 +74,14 @@ (D.prettyPrint (predict boxedClf)) (D.prettyPrint (predict packedClf)) -{- | A full outer join on a packed-text KEY column (exercises the-coalesceKeyColumn path that previously errored on PackedText). Result must-match the boxed-key join and recover all four distinct keys.--}+-- | A full outer join on a packed-text key matches the boxed-key join. fullOuterJoinOnTextKey :: Test fullOuterJoinOnTextKey = TestCase $ do let l mk =- D.fromNamedColumns+ D.fromColumns [("k", mk ["a", "b", "c"]), ("lv", DI.fromList ([1, 2, 3] :: [Double]))] r mk =- D.fromNamedColumns+ D.fromColumns [("k", mk ["b", "c", "d"]), ("rv", DI.fromList ([20, 30, 40] :: [Double]))] packedJoin = fullOuterJoin ["k"] (l packed) (r packed) boxedJoin = fullOuterJoin ["k"] (l boxed) (r boxed)
tests/Parquet.hs view
@@ -146,7 +146,7 @@ TestCase ( assertEqual "predicateWithOpts"- (D.fromNamedColumns [("id", D.fromList [6 :: Int32, 7])])+ (D.fromColumns [("id", D.fromList [6 :: Int32, 7])]) ( unsafePerformIO ( D.readParquetWithOpts ( D.defaultParquetReadOptions@@ -169,7 +169,7 @@ TestCase ( assertEqual "predicateUsesNonSelectedColumnWithOpts"- (D.fromNamedColumns [("bool_col", D.fromList [True, False])])+ (D.fromColumns [("bool_col", D.fromList [True, False])]) ( unsafePerformIO ( D.readParquetWithOpts ( D.defaultParquetReadOptions@@ -353,10 +353,6 @@ where value = 0xA1B2C3D4 :: Word32 --- ------------------------------------------------------------------------------ Group 1: Plain variant--- ---------------------------------------------------------------------------- allTypesTinyPagesPlain :: Test allTypesTinyPagesPlain = TestCase@@ -371,15 +367,7 @@ ) ) --- ------------------------------------------------------------------------------ Group 2: Compression codecs (unsupported → error tests)--- ------------------------------------------------------------------------------- TODO: LZ4 and LZ4_RAW compression are not yet implemented. When support--- is added via a Haskell lz4 binding, hadoopLz4Compressed,--- hadoopLz4CompressedLarger, nonHadoopLz4Compressed, lz4RawCompressed, and--- lz4RawCompressedLarger should all change from assertExpectException to--- assertEqual checking their respective row/column dimensions.+-- TODO: Once LZ4/LZ4_RAW are implemented, the lz4 tests become assertEqual. hadoopLz4Compressed :: Test hadoopLz4Compressed = TestCase@@ -425,9 +413,7 @@ (D.readParquet "./tests/data/lz4_raw_compressed_larger.parquet") ) --- Was: assertExpectException "concatenatedGzipMembers" "12" ...--- The old parser failed with a ZLIB size error. The new decompressor--- handles concatenated gzip members correctly.+-- The decompressor now handles concatenated gzip members. concatenatedGzipMembers :: Test concatenatedGzipMembers = TestCase@@ -442,9 +428,7 @@ ) ) --- TODO: BROTLI compression is not yet implemented. When a Haskell brotli--- binding is added, change this to assertEqual checking the actual--- dimensions of large_string_map.brotli.parquet.+-- TODO: BROTLI is not implemented; change this to assertEqual once it is. largeBrotliMap :: Test largeBrotliMap = TestCase@@ -454,18 +438,7 @@ (D.readParquet "./tests/data/large_string_map.brotli.parquet") ) --- ------------------------------------------------------------------------------ Group 3: Delta / RLE encodings (unsupported → error tests)--- ------------------------------------------------------------------------------- Was: assertExpectException "deltaBinaryPacked" "EDELTA_BINARY_PACKED" ...--- The new parser's error includes the encoding name "DELTA_BINARY_PACKED"--- without the old "E" prefix used in the previous error format.--- TODO: When DELTA_BINARY_PACKED (encoding id=5) is implemented, change--- this to assertEqual checking actual dimensions. The encoding stores--- integer data as bit-packed deltas and is common for monotonically--- increasing columns (row IDs, timestamps):--- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-encoding-delta_binary_packed--5+-- TODO: Once DELTA_BINARY_PACKED is implemented, assert actual dimensions. deltaBinaryPacked :: Test deltaBinaryPacked = TestCase@@ -475,12 +448,7 @@ (D.readParquet "./tests/data/delta_binary_packed.parquet") ) --- Was: assertExpectException "deltaByteArray" "EDELTA_BYTE_ARRAY" ...--- Same reason as deltaBinaryPacked: new error format drops the "E" prefix.--- TODO: When DELTA_BYTE_ARRAY (encoding id=7) is implemented, change this--- to assertEqual checking actual dimensions. The encoding prefix-differences--- consecutive string values, reducing storage for sorted byte arrays:--- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-strings-delta_byte_array--7+-- TODO: Once DELTA_BYTE_ARRAY is implemented, assert actual dimensions. deltaByteArray :: Test deltaByteArray = TestCase@@ -490,12 +458,7 @@ (D.readParquet "./tests/data/delta_byte_array.parquet") ) --- Was: assertExpectException "deltaEncodingOptionalColumn" "EDELTA_BINARY_PACKED" ...--- The first column that errors in this file uses DELTA_BYTE_ARRAY encoding,--- so we match the broader "unsupported encoding" substring instead.--- TODO: Once DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY are both implemented,--- change this to assertEqual checking the actual row count of--- delta_encoding_optional_column.parquet.+-- TODO: Once both DELTA encodings are implemented, assert the row count. deltaEncodingOptionalColumn :: Test deltaEncodingOptionalColumn = TestCase@@ -505,10 +468,7 @@ (D.readParquet "./tests/data/delta_encoding_optional_column.parquet") ) --- Was: assertExpectException "deltaEncodingRequiredColumn" "EDELTA_BINARY_PACKED" ...--- Same as deltaEncodingOptionalColumn: first failing column uses DELTA_BYTE_ARRAY.--- TODO: Same as deltaEncodingOptionalColumn — change to assertEqual once--- DELTA_BINARY_PACKED and DELTA_BYTE_ARRAY encodings are both supported.+-- TODO: Once both DELTA encodings are implemented, change to assertEqual. deltaEncodingRequiredColumn :: Test deltaEncodingRequiredColumn = TestCase@@ -518,14 +478,7 @@ (D.readParquet "./tests/data/delta_encoding_required_column.parquet") ) --- Was: assertExpectException "deltaLengthByteArray" "ZSTD" ...--- The old parser failed during ZSTD decompression. The new parser--- detects the unsupported DELTA_LENGTH_BYTE_ARRAY encoding before decompression.--- TODO: When DELTA_LENGTH_BYTE_ARRAY (encoding id=6) is implemented, change--- this to assertEqual checking actual dimensions. The encoding stores a--- delta-encoded list of byte-array lengths followed by the raw concatenated--- values:--- https://parquet.apache.org/docs/file-format/data-pages/encodings/#delta-length-byte-array-delta_length_byte_array--6+-- TODO: Once DELTA_LENGTH_BYTE_ARRAY is implemented, assert dimensions. deltaLengthByteArray :: Test deltaLengthByteArray = TestCase@@ -535,13 +488,7 @@ (D.readParquet "./tests/data/delta_length_byte_array.parquet") ) --- Was: assertExpectException "rleBooleanEncoding" "Zlib" ...--- The old parser failed during Zlib decompression. The new parser--- detects the unsupported RLE boolean encoding before reaching decompression.--- TODO: When RLE/Bit-Packing Hybrid (encoding id=3, bit-width=1) is--- implemented for BOOLEAN columns, change this to assertEqual checking the--- actual decoded boolean values. The encoding is spec-valid for BOOLEAN:--- https://parquet.apache.org/docs/file-format/data-pages/encodings/#run-length-encoding--bit-packing-hybrid-rle--3+-- TODO: Once RLE boolean encoding is implemented, assert the decoded values. rleBooleanEncoding :: Test rleBooleanEncoding = TestCase@@ -551,15 +498,7 @@ (D.readParquet "./tests/data/rle_boolean_encoding.parquet") ) --- Was: assertExpectException "dictPageOffsetZero" "Unknown kv" ...--- The old parser reported "Unknown kv" for a bad key-value field. The new--- Pinch-based page-header parser reports "Field 1 is absent" for the--- malformed page header in this file.--- TODO: Investigate whether dict-page-offset-zero.parquet can be read--- successfully with a more lenient page-header parser. If the missing--- mandatory field can be treated as a per-page soft error rather than--- aborting the whole read, this test would change to assertEqual--- checking actual dimensions.+-- TODO: Check whether a lenient page-header parser can read this file. dictPageOffsetZero :: Test dictPageOffsetZero = TestCase@@ -569,22 +508,7 @@ (D.readParquet "./tests/data/dict-page-offset-zero.parquet") ) --- ------------------------------------------------------------------------------ Group 4: Data Page V2 (unsupported → error tests)--- ------------------------------------------------------------------------------- Was: assertExpectException "datapageV2Snappy" "InvalidOffset" ...--- The old parser failed with an offset validation error. The new parser--- first encounters the unsupported RLE encoding used by data-page-v2.--- TODO: Full Data Page V2 support requires two changes:--- 1. RLE/Bit-Packing Hybrid (id=3, bit-width=1) for BOOLEAN values--- (shared with rleBooleanEncoding above).--- 2. Parsing DataPageHeaderV2's in-line level streams: in v2, definition--- and repetition levels are stored uncompressed before the (optionally--- compressed) value bytes, with lengths given by--- definition_levels_byte_length and repetition_levels_byte_length.--- Once both are done, change to assertEqual checking actual dimensions:--- https://parquet.apache.org/docs/file-format/data-pages/+-- TODO: Data Page V2 needs RLE booleans and in-line level stream parsing. datapageV2Snappy :: Test datapageV2Snappy = TestCase@@ -594,19 +518,7 @@ (D.readParquet "./tests/data/datapage_v2.snappy.parquet") ) --- Was: assertExpectException "datapageV2EmptyDatapage" "UnexpectedEOF" ...--- The old Snappy decompressor raised "UnexpectedEOF". The new Snappy--- library raises "EmptyInput" when given zero-length compressed data.--- The v2 page structure is parsed correctly: readLevelsV2V strips the--- in-line level streams before decompression, leaving an empty value--- payload (BS.empty) for a page with 0 values. The Snappy decompressor--- then raises "EmptyInput" because it is handed zero bytes.--- TODO: An empty data page (0 values) is valid and should contribute--- 0 rows without raising an error. The fix is a single guard in the--- DATA_PAGE_V2 branch of readPages (Page.hs): short-circuit--- decompressData when compValBytes is empty, returning BS.empty--- directly. Once fixed, change this to assertEqual checking the--- total expected row count of the file.+-- TODO: Guard empty V2 data pages in readPages so they yield 0 rows. datapageV2EmptyDatapage :: Test datapageV2EmptyDatapage = TestCase@@ -616,9 +528,7 @@ (D.readParquet "./tests/data/datapage_v2_empty_datapage.snappy.parquet") ) --- Was: assertExpectException "pageV2EmptyCompressed" "10" ...--- The old parser failed on empty compressed page-v2 blocks. The new parser--- treats empty compressed data as zero-value pages and reads all 10 rows.+-- Empty compressed page-v2 data is treated as zero-value pages. pageV2EmptyCompressed :: Test pageV2EmptyCompressed = TestCase@@ -633,10 +543,6 @@ ) ) --- ------------------------------------------------------------------------------ Group 5: Checksum files (all read successfully)--- ---------------------------------------------------------------------------- datapageV1UncompressedChecksum :: Test datapageV1UncompressedChecksum = TestCase@@ -693,12 +599,7 @@ ) ) --- TODO: CRC checksum validation is not yet implemented; corrupt page--- checksums are silently ignored. When validation is added, consider a--- validateChecksums :: Bool field in ParquetReadOptions (default False)--- so callers can opt in. Once implemented, datapageV1CorruptChecksum and--- rleDictUncompressedCorruptChecksum should change to assertExpectException--- checking for a checksum mismatch error.+-- TODO: CRC checksum validation is not implemented; corrupt ones are ignored. datapageV1CorruptChecksum :: Test datapageV1CorruptChecksum = TestCase@@ -727,10 +628,6 @@ ) ) --- ------------------------------------------------------------------------------ Group 6: NULL handling--- ---------------------------------------------------------------------------- nullsSnappy :: Test nullsSnappy = TestCase@@ -797,10 +694,6 @@ ) ) --- ------------------------------------------------------------------------------ Group 7: Decimal types--- ---------------------------------------------------------------------------- int32Decimal :: Test int32Decimal = TestCase@@ -823,6 +716,19 @@ ) ) +nullableIntDecimal :: Test+nullableIntDecimal = testBothReadParquetPaths $ \readParquet ->+ TestCase+ ( assertEqual+ "nullableIntDecimal"+ ( D.fromColumns+ [ ("int32_dec", D.fromList [Just (1 :: Double), Nothing, Just 2.5, Just 0.75])+ , ("int64_dec", D.fromList [Nothing, Just (1234.5 :: Double), Nothing, Just (-5)])+ ]+ )+ (unsafePerformIO (readParquet "./tests/data/nullable_int_decimal.parquet"))+ )+ byteArrayDecimal :: Test byteArrayDecimal = TestCase@@ -834,16 +740,7 @@ ) ) --- Was: assertExpectException "fixedLengthDecimal" "FIXED_LEN_BYTE_ARRAY" ...--- The old parser recognised FIXED_LEN_BYTE_ARRAY as a physical type but--- had no page decoder for it; reading data from such a column threw an--- error at the decoding stage. The new parser's fixedLenByteArrayDecoder--- reads the raw bytes and surfaces them as a text column.--- TODO: When the DECIMAL logical type is properly decoded for--- FIXED_LEN_BYTE_ARRAY columns, replace this dimension-only check with a--- value-level assertion verifying the actual decimal values (e.g. as--- Scientific or Double). The raw-byte Text column should become a typed--- numeric column.+-- TODO: Once DECIMAL is decoded for FIXED_LEN_BYTE_ARRAY, assert values. fixedLengthDecimal :: Test fixedLengthDecimal = TestCase@@ -855,11 +752,7 @@ ) ) --- Was: assertExpectException "fixedLengthDecimalLegacy" "FIXED_LEN_BYTE_ARRAY" ...--- Same as fixedLengthDecimal: the old parser had no page decoder for--- FIXED_LEN_BYTE_ARRAY; the new parser's fixedLenByteArrayDecoder handles it.--- TODO: Same as fixedLengthDecimal — add a value-level assertion once--- DECIMAL decoding over FIXED_LEN_BYTE_ARRAY is implemented.+-- TODO: Add a value-level assertion once DECIMAL decoding is implemented. fixedLengthDecimalLegacy :: Test fixedLengthDecimalLegacy = TestCase@@ -874,10 +767,6 @@ ) ) --- ------------------------------------------------------------------------------ Group 8: Binary / fixed-length bytes--- ---------------------------------------------------------------------------- binaryFile :: Test binaryFile = TestCase@@ -903,9 +792,7 @@ ) ) --- Was: assertExpectException "fixedLengthByteArray" "FIXED_LEN_BYTE_ARRAY" ...--- Same as fixedLengthDecimal: the old parser had no page decoder for--- FIXED_LEN_BYTE_ARRAY; the new parser's fixedLenByteArrayDecoder handles it.+-- FIXED_LEN_BYTE_ARRAY is read by fixedLenByteArrayDecoder as raw text. fixedLengthByteArray :: Test fixedLengthByteArray = TestCase@@ -917,10 +804,6 @@ ) ) --- ------------------------------------------------------------------------------ Group 9: INT96 timestamps--- ---------------------------------------------------------------------------- int96FromSpark :: Test int96FromSpark = TestCase@@ -932,13 +815,7 @@ ) ) --- ------------------------------------------------------------------------------ Group 10: Metadata / index / bloom filters--- ------------------------------------------------------------------------------- Was: assertExpectException "columnChunkKeyValueMetadata" "Unknown page header field" ...--- The old parser rejected extra fields in page headers. Pinch ignores--- unknown fields gracefully. This file contains 0 data rows.+-- Unknown page header fields are ignored; this file has 0 data rows. columnChunkKeyValueMetadata :: Test columnChunkKeyValueMetadata = TestCase@@ -981,15 +858,7 @@ ) ) --- Was: assertEqual "sortColumns" (3, 2) ...--- The file contains two row groups, each storing 3 rows (6 rows total).--- DuckDB's parquet-metadata output shows row_group_num_rows=3, which is--- the count *per row group*, not the file total.row group*, not the file total.row group*, not the file total.row group*, not the file total.--- https://github.com/apache/parquet-testing/blob/master/data/README.md#:~:text=sort_columns.parquet--- The above link is to the repository the test parquet files comes from.--- The table describes sort_columns.parquet as having two row groups.--- The old parser only read the first row group (a bug). The new parser--- reads all row groups and returns (6, 2) correctly.+-- Two row groups of 3 rows each: all row groups are read, giving (6, 2). sortColumns :: Test sortColumns = TestCase@@ -1001,9 +870,7 @@ ) ) --- Was: assertExpectException "overflowI16PageCnt" "UNIMPLEMENTED" ...--- The old parser used Int16 for page counts and overflowed on this file.--- The new parser uses Int32 and reads all 40,000 rows correctly.+-- The parser uses Int32 page counts and reads all 40,000 rows. overflowI16PageCnt :: Test overflowI16PageCnt = TestCase@@ -1015,18 +882,7 @@ ) ) --- ------------------------------------------------------------------------------ Group 11: Nested / complex types and byte-stream-split--- ------------------------------------------------------------------------------- Was: assertExpectException "byteStreamSplitZstd" "EBYTE_STREAM_SPLIT" ...--- The new parser's error includes the encoding name "BYTE_STREAM_SPLIT"--- without the old "E" prefix used in the previous error format.--- TODO: When BYTE_STREAM_SPLIT (encoding id=9) is implemented, change this--- to assertEqual checking actual dimensions. The encoding interleaves the--- individual byte streams of multi-byte scalars to improve compression for--- floating-point and other structured data:--- https://parquet.apache.org/docs/file-format/data-pages/encodings/#byte-stream-split-byte_stream_split--9+-- TODO: Once BYTE_STREAM_SPLIT is implemented, assert actual dimensions. byteStreamSplitZstd :: Test byteStreamSplitZstd = TestCase@@ -1036,13 +892,7 @@ (D.readParquet "./tests/data/byte_stream_split.zstd.parquet") ) --- Was: assertExpectException "byteStreamSplitExtendedGzip" "FIXED_LEN_BYTE_ARRAY" ...--- The old parser had no page decoder for FIXED_LEN_BYTE_ARRAY and threw--- before ever inspecting the encoding. The new parser handles the physical--- type but the BYTE_STREAM_SPLIT encoding used for values is not yet--- implemented, so the error message shifts from the type to the encoding.--- TODO: Same as byteStreamSplitZstd — change to assertEqual once--- BYTE_STREAM_SPLIT encoding is supported.+-- TODO: Change to assertEqual once BYTE_STREAM_SPLIT is supported. byteStreamSplitExtendedGzip :: Test byteStreamSplitExtendedGzip = TestCase@@ -1052,18 +902,7 @@ (D.readParquet "./tests/data/byte_stream_split_extended.gzip.parquet") ) --- Was: assertExpectException "float16NonzerosAndNans" "PFIXED_LEN_BYTE_ARRAY" ...--- The "PFIXED_LEN_BYTE_ARRAY" in the old error was the Show of the old--- parser's ParquetType enum hitting a catch-all dispatch branch — it--- recognised the physical type but had no decoder for it. The new parser's--- fixedLenByteArrayDecoder reads 2-byte FIXED_LEN_BYTE_ARRAY (float16)--- columns as raw-byte text; proper float16 value decoding is not yet--- implemented.--- TODO: When IEEE 754 half-precision (float16) decoding is implemented,--- add a value-level assertion using hasElemType @Float (or a dedicated--- Float16 type if one is introduced). Verify that the decoded values match--- the known reference values for float16_nonzeros_and_nans.parquet.--- The column should no longer be exposed as raw-byte Text.+-- TODO: Once float16 decoding is implemented, assert the decoded values. float16NonzerosAndNans :: Test float16NonzerosAndNans = TestCase@@ -1078,11 +917,7 @@ ) ) --- Was: assertExpectException "float16ZerosAndNans" "PFIXED_LEN_BYTE_ARRAY" ...--- Same as float16NonzerosAndNans: old parser had no decoder for the--- FIXED_LEN_BYTE_ARRAY physical type; new parser reads raw bytes as text.--- TODO: Same as float16NonzerosAndNans — add a value-level assertion once--- float16 decoding is implemented.+-- TODO: Add a value-level assertion once float16 decoding is implemented. float16ZerosAndNans :: Test float16ZerosAndNans = TestCase@@ -1207,14 +1042,7 @@ ) ) --- Was: assertExpectException "unknownLogicalType" "Unknown logical type" ...--- The old parser raised a custom "Unknown logical type" message. The new--- Pinch-based metadata parser raises "Field 16 is absent" for the--- unrecognised LogicalType variant in this file.--- TODO: If Pinch is extended to support forward-compatible decoding of--- unknown union variants (treating unrecognised logical-type IDs as absent--- rather than raising an error), change this to assertEqual where the file--- parses successfully and the column falls back to its physical type.+-- TODO: If Pinch decodes unknown union variants, assert a successful parse. unknownLogicalType :: Test unknownLogicalType = TestCase@@ -1224,19 +1052,7 @@ (D.readParquet "./tests/data/unknown-logical-type.parquet") ) --- ------------------------------------------------------------------------------ Group 12: Malformed files--- ------------------------------------------------------------------------------- Was: assertExpectException "nationDictMalformed" "dict index count mismatch" ...--- The old parser validated the dictionary entry count against data-page--- indices and raised "dict index count mismatch". The new parser does not--- replicate that check; the dictionary bytes happen to decode correctly--- despite the metadata discrepancy, returning the complete 25-row dataset.--- TODO: If a stricter dictionary-validation pass is added (checking that--- the number of decoded entries matches num_values in the dictionary page--- header), revert this to assertExpectException with a count-mismatch--- substring.+-- TODO: If stricter dictionary validation is added, expect a count mismatch. nationDictMalformed :: Test nationDictMalformed = TestCase@@ -1308,17 +1124,13 @@ "score should NOT be nullable" (not (hasMissing (unsafeGetColumn "score" df))) -{- | Nanosecond-precision timestamps (@TIMESTAMP(NANOS)@) must decode to the-correct 'UTCTime'. Regression test for the unit-scaling bug where the NANOS-multiplier @1_000_000 \`div\` 1_000_000_000@ truncated to 0, collapsing every-value to the epoch.--}+-- | Nanosecond timestamps must decode to the correct UTCTime. timestampNanos :: Test timestampNanos = testBothReadParquetPaths $ \readParquet -> TestCase $ assertEqual "TIMESTAMP(NANOS) decodes to correct UTCTime"- ( D.fromNamedColumns+ ( D.fromColumns [ ( "ts" , D.fromList@@ -1394,6 +1206,7 @@ , -- Group 7: decimal types int32Decimal , int64Decimal+ , nullableIntDecimal , byteArrayDecimal , fixedLengthDecimal , fixedLengthDecimalLegacy
tests/ParquetTestData.hs view
@@ -1,9 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Large DataFrame fixtures used by the Parquet test suite.-Kept in a separate module to avoid cluttering Parquet.hs.--}+-- | Large DataFrame fixtures used by the Parquet test suite. module ParquetTestData ( allTypes, tinyPagesLast10,@@ -18,7 +16,7 @@ allTypes :: D.DataFrame allTypes =- D.fromNamedColumns+ D.fromColumns [ ("id", D.fromList [4 :: Int32, 5, 6, 7, 2, 3, 0, 1]) , ("bool_col", D.fromList [True, False, True, False, True, False, True, False]) , ("tinyint_col", D.fromList [0 :: Int32, 1, 0, 1, 0, 1, 0, 1])@@ -58,7 +56,7 @@ tinyPagesLast10 :: D.DataFrame tinyPagesLast10 =- D.fromNamedColumns+ D.fromColumns [ ("id", D.fromList @Int32 (reverse [6174 .. 6183])) , ("bool_col", D.fromList @Bool (take 10 (cycle [False, True]))) , ("tinyint_col", D.fromList @Int32 [3, 2, 1, 0, 9, 8, 7, 6, 5, 4])@@ -121,7 +119,7 @@ transactions :: D.DataFrame transactions =- D.fromNamedColumns+ D.fromColumns [ ("transaction_id", D.fromList [1 :: Int32, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]) , ( "event_time"@@ -279,7 +277,7 @@ mtCarsDataset :: D.DataFrame mtCarsDataset =- D.fromNamedColumns+ D.fromColumns [ ( "model" , D.fromList
tests/Plotting.hs view
@@ -5,11 +5,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeApplications #-} -{- |-Tests for the Vega-Lite web plotting backend: field-type inference, the-box-plot mark fix, NaN handling, escaping, computed-expression encodings, and-typed/untyped spec parity.--}+-- | Tests for the Vega-Lite plotting backend, including typed parity. module Plotting (tests) where import Data.Aeson (Value (Array, Null, Object, String), toJSON)@@ -30,24 +26,19 @@ import qualified DataFrame.Display.Web.Chart.Typed as CT import qualified DataFrame.Display.Web.Plot as P --- Unqualified so the record update below has a single 'includeZero' field--- label in scope; GHC < 9.8 can't disambiguate the qualified duplicate field.+-- Unqualified: GHC < 9.8 can't disambiguate the qualified duplicate field. import DataFrame.Display.Web.Plot (Scatter (includeZero)) --- ------------------------------------------------------------------------------ Fixtures + JSON helpers--- ---------------------------------------------------------------------------- numFrame :: D.DataFrame numFrame =- D.fromNamedColumns+ D.fromColumns [ ("a", D.fromList ([1.0, 2.0, 3.0, 4.0] :: [Double])) , ("b", D.fromList ([10.0, 20.0, 30.0, 40.0] :: [Double])) ] mixedFrame :: D.DataFrame mixedFrame =- D.fromNamedColumns+ D.fromColumns [ ("a", D.fromList ([1.0, 2.0, 3.0] :: [Double])) , ("g", D.fromList (["x", "y", "x"] :: [T.Text])) ]@@ -64,10 +55,6 @@ Just (Array xs) -> xs _ -> V.empty --- ------------------------------------------------------------------------------ Test cases--- ---------------------------------------------------------------------------- fieldTypeInference :: Test fieldTypeInference = TestCase $ do let spec =@@ -107,7 +94,7 @@ nanBecomesNull :: Test nanBecomesNull = TestCase $ do- let df = D.fromNamedColumns [("a", D.fromList ([0 / 0, 1.0] :: [Double]))]+ let df = D.fromColumns [("a", D.fromList ([0 / 0, 1.0] :: [Double]))] spec = C.toVegaSpec (C.chart df & C.enc C.Y (col @Double "a")) firstA = lookupKey "a" (fromMaybe Null (dataValues spec V.!? 0)) assertEqual "NaN inlines as null" (Just Null) firstA@@ -115,7 +102,7 @@ escapingSafe :: Test escapingSafe = TestCase $ do let weird = "we\"ir\\d"- df = D.fromNamedColumns [(weird, D.fromList ([1.0, 2.0] :: [Double]))]+ df = D.fromColumns [(weird, D.fromList ([1.0, 2.0] :: [Double]))] spec = C.toVegaSpec (C.chart df & C.enc C.X (col @Double weird)) row0 = fromMaybe Null (dataValues spec V.!? 0) assertBool
tests/PrettyPrint.hs view
@@ -1,10 +1,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Golden tests pinning the format of 'prettyPrint'/'prettyPrintWidth': flat-@else if@ ladders, operator-precedence parenthesization, width-aware wrapping of-commutative chains, and short expressions staying on one line.--}+-- | Golden tests pinning the format of prettyPrint and prettyPrintWidth. module PrettyPrint (tests) where import qualified Data.Text as T@@ -42,6 +39,15 @@ (ifThenElse (col @Double "b" .>. lit 2.0) (lit "y") (lit "z")) ) )+ , golden+ "caseWhen chain is a flat ladder"+ "if a .>. 1.0\n \"x\"\nelse if b .>. 2.0\n \"y\"\nelse\n \"z\""+ ( prettyPrint+ ( caseWhen (col @Double "a" .>. lit 1.0) (lit @T.Text "x") $+ caseWhen (col @Double "b" .>. lit 2.0) (lit "y") $+ orElse (lit "z")+ )+ ) , -- A lower-precedence operand is parenthesized under a higher one. golden "precedence: (a + b) * c" "(a + b) * c" (prettyPrint ((a + b) * c)) , -- No spurious parens when precedence already disambiguates.@@ -63,7 +69,7 @@ "boolean chain fits: no paren noise" "a + b .>=. c .&&. a .>. b" (prettyPrintWidth 80 ((a + b .>=. c) .&&. (a .>. b)))- , -- When a nested operand wraps, parens delimit it; flat operands stay bare.+ , -- When a nested operand wraps, parens delimit it; flat ones stay bare. golden "wrapped operand gets parens" "(a + b + c\n .>=. 0.0)\n .&&. a .>. b"
tests/Properties/Categorical.hs view
@@ -1,18 +1,6 @@ {-# LANGUAGE OverloadedStrings #-} -{- |-Property tests pinning the categorical laws the library is meant to obey,-following https://mchav.github.io/what-category-theory-teaches-us-about-dataframes/-- * /Topos (set algebra)/ — 'D.union', 'D.intersect', 'D.difference' and- 'D.symmetricDifference' form the subobject lattice, with 'D.distinct'- (image factorization) as the canonical set-valued map.- * /Migration functor Δ/ — 'D.select'/'D.rename'/'D.exclude' are functorial:- identities and round-trips hold.--Operands that must share a schema are produced by row-subsetting one generated-base DataFrame, so the two/three frames are always schema-compatible.--}+-- | Property tests pinning the categorical laws of set algebra and migration. module Properties.Categorical (tests) where import Data.Text ()@@ -30,11 +18,7 @@ nRows :: DataFrame -> Int nRows = fst . dataframeDimensions -{- | Equality of the underlying row sets, via the library's null-aware-@Eq DataFrame@. Set operations emit rows in a deterministic hash-bucket order-that depends only on content, and 'D.Eq' compares null slots correctly, so this-is a sound oracle for the topos laws (including over @Maybe Int@ columns).--}+-- | Equality of the underlying row sets, a sound oracle for the topos laws. sameRows :: DataFrame -> DataFrame -> Property sameRows x y = x === y @@ -44,20 +28,14 @@ -- | A schema-compatible triple, likewise. data Triple = Triple DataFrame DataFrame DataFrame deriving (Show) -{- | A base frame of @Int@ and @Maybe Int@ columns.--@Maybe Int@ is included so the laws exercise nulls — @Nothing@ must behave as a-value distinct from any @Just n@. @Double@ is deliberately excluded: @NaN@/@-0.0@-break set semantics by IEEE rules (@NaN /= NaN@), which is not a library bug. A-small value range makes duplicate rows common, exercising deduplication.--}+-- | A base frame of Int and Maybe Int columns, with many duplicate rows. genBase :: Gen DataFrame genBase = do nCols <- choose (1, 4) nRowsG <- choose (0, 60) let names = take nCols ["c0", "c1", "c2", "c3"] cols <- mapM (const (genCol nRowsG)) names- pure (D.fromNamedColumns (zip names cols))+ pure (D.fromColumns (zip names cols)) where genCol n = oneof@@ -87,12 +65,7 @@ base <- genBase Triple <$> subset base <*> subset base <*> subset base -{- | A single clean base frame (Int / Maybe Int columns only).--Used by the Δ-functor laws, which compare with the representation-sensitive-@Eq DataFrame@ — and that @Eq@ is not even reflexive on @NaN@, so the shared-@Double@-bearing generator cannot be used here.--}+-- | A single clean base frame (Int and Maybe Int columns only). newtype Frame = Frame DataFrame deriving (Show) instance Arbitrary Frame where@@ -102,13 +75,6 @@ emptyLike :: DataFrame -> DataFrame emptyLike = D.range (0, 0) ----------------------------------------------------------------------------------- Topos / set-algebra laws------ These are statements about row /sets/, so they are asserted with 'sameRows'--- (row-content equality) rather than the representation-sensitive @Eq DataFrame@.--------------------------------------------------------------------------------- prop_unionCommutative :: Pair -> Property prop_unionCommutative (Pair a b) = sameRows (D.union a b) (D.union b a) @@ -151,10 +117,6 @@ prop_differenceDisjoint :: Pair -> Property prop_differenceDisjoint (Pair a b) = nRows (D.intersect (D.difference a b) b) === 0------------------------------------------------------------------------------------ Δ migration functor laws-------------------------------------------------------------------------------- -- | Excluding nothing is the identity. prop_excludeNothingIdentity :: Frame -> Property
tests/Simplify.hs view
@@ -2,9 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Specification for 'DataFrame.Internal.Expression.Simplify.simplify': each case is the-full predicate expression, compared with 'eqExpr'.--}+-- | Specification for simplify: each case compares the full expression. module Simplify (tests) where import DataFrame.Expression.Operators
tests/Typed/IOReaders.hs view
@@ -2,11 +2,7 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-} -{- | Tests for the typed CSV reader: a write\/read round-trip matches the untyped-reader, a wrong schema surfaces as 'Left' via 'readCsvWithError', and the-throwing 'readCsv' raises a 'DataFrameException'. The Parquet reader uses the-identical freeze-on-read path.--}+-- | Tests for the typed CSV reader: round-trip, wrong schema, throwing reads. module Typed.IOReaders (tests) where import Control.Exception (evaluate, try)@@ -32,7 +28,7 @@ sampleDF :: D.DataFrame sampleDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList [1, 2, 3 :: Int]) , ("y", DI.fromList [1.5, 2.5, 3.5 :: Double]) , ("g", DI.fromList ["a", "b", "c" :: T.Text])@@ -62,9 +58,7 @@ IO (Either DataFrameException Int) assertBool "wrong schema => throws DataFrameException" (isLeft r) -{- | The schema is the read specification: only its columns are fetched, out-of a file that holds more.--}+-- | The schema is the read specification: only its columns are fetched. typedReadProjects :: Test typedReadProjects = TestCase $ withSystemTempFile "typed_proj.csv" $ \fp h -> do hClose h@@ -73,9 +67,7 @@ assertEqual "only the schema's column is read" ["g"] (D.columnNames narrow) assertEqual "rows intact" (3, 1) (D.dimensions narrow) -{- | The schema's types win over inference: a column of digits declared 'Text'-comes back as 'Text', which a plain read would have called 'Int'.--}+-- | The schema's types win over inference. typedReadSharesTypes :: Test typedReadSharesTypes = TestCase $ withSystemTempFile "typed_types.csv" $ \fp h -> do hClose h@@ -91,7 +83,7 @@ (Just (DI.fromList ["1", "2", "3" :: T.Text])) (getColumn "n" typed) where- digits = D.fromNamedColumns [("n", DI.fromList [1, 2, 3 :: Int])]+ digits = D.fromColumns [("n", DI.fromList [1, 2, 3 :: Int])] tests :: [Test] tests =
tests/Typed/Parity.hs view
@@ -25,7 +25,7 @@ baseDF :: D.DataFrame baseDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList [1, 2, 3, 4, 5, 6 :: Int]) , ("y", DI.fromList [10.0, 25.0, 30.0, 5.0, 50.0, 12.0 :: Double]) , ("g", DI.fromList ["a", "b", "a", "b", "a", "b" :: T.Text])@@ -173,10 +173,10 @@ -- Disjoint frames for the (|||) test. leftDF :: D.DataFrame-leftDF = D.fromNamedColumns [("x", DI.fromList [1, 2 :: Int])]+leftDF = D.fromColumns [("x", DI.fromList [1, 2 :: Int])] rightDF :: D.DataFrame-rightDF = D.fromNamedColumns [("y", DI.fromList [1.0, 2.0 :: Double])]+rightDF = D.fromColumns [("y", DI.fromList [1.0, 2.0 :: Double])] leftT :: DT.TypedDataFrame '[ '("x", Int)] leftT = either (error . show) id (DT.freezeWithError leftDF)@@ -187,7 +187,7 @@ -- Numeric-only frame for the matrix test. numericDF :: D.DataFrame numericDF =- D.fromNamedColumns+ D.fromColumns [ ("x", DI.fromList [1, 2, 3 :: Int]) , ("y", DI.fromList [1.5, 2.5, 3.5 :: Double]) ]
+ tests/data/nullable_int_decimal.parquet view
binary file changed (absent → 812 bytes)