dataframe 3.7.0.0 → 3.8.0.0

raw patch · 56 files changed

+2136/−677 lines, 56 filesdep ~dataframedep ~dataframe-coredep ~dataframe-csvPVP ok

version bump matches the API change (PVP)

Dependency ranges changed: dataframe, dataframe-core, dataframe-csv, dataframe-expr-serializer, dataframe-json, dataframe-lazy, dataframe-operations, dataframe-parquet, dataframe-viz

API changes (from Hackage documentation)

Files

• CHANGELOG.md +93/−0
• README.md +37/−40
• app/Benchmark.hs +3/−4
• app/LazyBenchmark.hs +5/−5
• app/Synthesis.hs +7/−6
• benchmark/Main.hs +2/−3
• dataframe.cabal +41/−34
• src/DataFrame.hs +2/−2
• src/DataFrame/Typed.hs +199/−75
• src/DataFrame/Typed/Interop.hs +63/−0
• tests/Functions.hs +3/−5
• tests/IR/ExprJsonRoundtrip.hs +29/−23
• tests/Internal/PackedText.hs +11/−11
• tests/LazyParity.hs +127/−18
• tests/LazyParquet.hs +21/−11
• tests/LazyProjection.hs +10/−10
• tests/Learn/Denotation.hs +3/−3
• tests/Learn/Ensembles.hs +20/−20
• tests/Learn/Metamorphic.hs +7/−7
• tests/Learn/MetricsTests.hs +6/−7
• tests/Learn/Models.hs +15/−15
• tests/Learn/Segmented.hs +16/−16
• tests/Learn/SklearnParity.hs +8/−8
• tests/Learn/Synthesis.hs +8/−9
• tests/Learn/TypedModel.hs +4/−4
• tests/Main.hs +4/−0
• tests/Monad.hs +2/−2
• tests/Operations/Aggregations.hs +22/−21
• tests/Operations/Apply.hs +3/−3
• tests/Operations/CountDistinct.hs +4/−5
• tests/Operations/Derive.hs +5/−4
• tests/Operations/ExprParse.hs +6/−6
• tests/Operations/Filter.hs +9/−8
• tests/Operations/InterpreterKernels.hs +4/−4
• tests/Operations/Join.hs +14/−13
• tests/Operations/Nullable.hs +34/−34
• tests/Operations/OperatorIdentity.hs +258/−0
• tests/Operations/ParallelGroupBy.hs +10/−10
• tests/Operations/Provenance.hs +11/−11
• tests/Operations/Record.hs +11/−11
• tests/Operations/RowWise.hs +6/−6
• tests/Operations/SetOps.hs +1/−2
• tests/Operations/Shuffle.hs +1/−1
• tests/Operations/Sort.hs +9/−10
• tests/Operations/SortMask.hs +5/−5
• tests/Operations/Subset.hs +5/−5
• tests/Operations/TextKernels.hs +2/−3
• tests/Operations/TopBy.hs +5/−6
• tests/Operations/VectorKernel.hs +3/−3
• tests/Operations/Window.hs +29/−29
• tests/PackedTextMigration.hs +4/−5
• tests/Parity/FourWay.hs +646/−0
• tests/Parquet.hs +5/−5
• tests/Plotting.hs +49/−2
• tests/Simplify.hs +104/−105
• tests/Typed/Parity.hs +125/−22

@@ -1,7 +1,100 @@ # Revision history for dataframe +## 3.8.0.0++### Breaking changes+* `filter` takes a boolean expression (`Expr Bool -> DataFrame -> DataFrame`),+  matching the lazy APIs. `filterWhere` is removed; use `filter`.+* The old `filter :: Expr a -> (a -> Bool) -> …` is removed; use `filterBy`,+  which takes the predicate first. The same applies to the typed API.+* `DataFrame.Monad.filterWhereM` is renamed to `filterM`.+* `DataFrame.Typed` mirrors `DataFrame`: expression combinators (`sum`,+  `mean`, `count`, `lift`, `not`, casts, `asc`/`desc`, …) move to the new+  `DataFrame.Typed.Functions`, imported like `DataFrame.Functions`.+  `DataFrame.Typed` keeps `col`, `lit`, conditionals and operators, and now+  exports the frame statistics (`mean`, `summarize`, `correlation`, …).+* `DataFrame.Lazy` takes the eager shapes: `groupBy keys` then+  `aggregate aggs`; `join jt [keys]` (multi-key) plus `innerJoin`,+  `leftJoin`, `rightJoin`, `fullOuterJoin`; `sortBy` takes the eager+  `SortOrder` (`Asc`/`Desc` expressions). `runDataFrame` is renamed `run`.+  The lazy `SortOrder (Ascending | Descending)` type is removed.+  `DataFrame.Lazy` re-exports the expression operators like `DataFrame`.+* `DataFrame.Typed.Lazy` mirrors `DataFrame.Typed`: `sortBy` takes+  `TSortOrder`, joins take a key list, `aggregate` takes an `as` builder, and+  scans read their schema from the type (`scanCsv @Schema path`).+  `fromTypedDataFrame` is removed (use `fromDataFrame`); expression+  combinators come from `DataFrame.Typed.Functions`.+* Typed eager `innerJoin` and `fullOuterJoin` and lazy INNER/FULL_OUTER joins+  put left columns before right ones, matching eager joins and the typed+  join schemas.+* Typed `applyColumn` is renamed `apply`; typed `fromRecordsTyped` /+  `toRecordsTyped` are exported from `DataFrame.Typed` as `fromRecords` /+  `toRecords`.+* Operators are identified by tag, not name. `UnaryOp`/`BinaryOp` expose+  `unaryId`/`binaryId :: OpRef …` (`Builtin` tag, `Named` text or+  `Anonymous` label); `unaryName`/`binaryName` are now plain functions.+  `MkUnaryOp`/`MkBinaryOp` take an `OpRef` instead of a name, and+  `AggStrategy` constructors take an `OpRef AggId`.+* `liftDecorated`/`lift2Decorated` build `Named` ops, which never use a+  builtin kernel; library code uses the new `liftBuiltin`/`lift2Builtin`.+* `CastWith`/`CastExprWith` take an `OpRef CastId` instead of a text tag.+* `kExtremes` takes a `Bool` (largest/smallest) instead of a tag and a+  comparator, so the two can no longer disagree.+* Every `DataFrame.Functions` combinator that does not take a Haskell+  function is a builtin; `lift`, `lift2`, `nullLift`, `nullLift2`,+  `reduce`, `whenPresent`, `whenBothPresent` and `bind` stay anonymous.+* Typed `not` prints as `~`, like the untyped `not`.+* `DataFrame.Functions` holds only expression combinators: `sanitize`,+  `isHaskellIdentifier`, `isReservedId` and `isVarId` move to+  `DataFrame.TH.Identifier` (dataframe-th), and `optionalToDoubleVector` to+  `DataFrame.Internal.Statistics`.+* `dataframe-viz`: typed charts reject nullable columns at compile time;+  `enc`, `facet`, `histogram`, `box`, etc. require `a ~ BaseType a`.+* Column shorthands in `DataFrame.Functions`: `int`, `double`,+  `text`, `bool`, their `maybe*` forms, and `int32`, `int64`, `float32`,+  `float64` (e.g. `F.double "x"` for `F.col @Double "x"`).+* `dataframe-viz`: `displayRows n` prints a frame showing at most `n` rows.+* `DataFrame.Lazy` and `DataFrame.Typed.Lazy` cover the eager API: row ops+  (`filterBy`, `filterJust`, `drop`, `takeLast`, `range`, `topBy`,+  `distinct`, `sample`, `shuffle`, splits and folds), column ops (`exclude`,+  `rename`, `deriveMany`, `impute`, `apply*`, `insert*`, `cloneColumn`, …),+  merges and set operations (`<>`/`append`, `|||`, `union`, `intersect`,+  `difference`, `symmetricDifference`), summaries, and value-returning ops+  (`mean`, `nRows`, `columnAs*`, `toCsv`, `writeCsv`, `writeParquet`,+  `display`, …) that run the query and return `IO`. Row-local ops stream+  batch by batch; ops that need the whole input collect it first.+* `joinOn jt [(leftKey, rightKey)]` in all four APIs (typed:+  `joinOn @'INNER @'[ '("id", "uid")]`); the result keeps the left names.+* Typed parity with the untyped API: `topBy`, `deriveMany` (built with+  `as`), `join @'INNER @'["key"]`, `applyInt`, `applyDouble`, `fold`,+  `empty`, `null`, `(|>)`, display/rendering (`display`, `toCsv`,+  `toMarkdown`, …), and IO re-exports from `DataFrame.Typed`.+* Typed `writeParquet`, `writeParquetWithOptions`,+  `readParquetFilesWithOpts`, `readSeparated`, `writeSeparated`, and the new+  `DataFrame.Typed.IO.JSON` (`readJSON`, `readJSONEither`).+* `DataFrame.Typed.Functions` gains `cast`, `castWithDefault`, `castEither`,+  `unsafeCast` (schema-checked column), `eq`, `lt`, `gt`, `leq`, `geq`,+  `and`, `or`, `countDistinct`, `topK`, `bottomK`, `isNull`, `isNotNull`,+  `contains` and `length`, completing parity with `DataFrame.Functions`.+* Lazy CSV and Parquet scans return columns in file order, like eager+  reads, instead of alphabetical order.+* A user function named like a builtin (e.g. `"add"`) no longer runs the+  builtin's kernel instead of the user's function.+* `eqExpr` no longer equates different `lift`/`lift2` functions, `fromMaybe`+  with different defaults, or `(^)` with `(^^)`.+* The predicate simplifier no longer merges conditions on different+  user functions, and no longer treats `fromMaybe True` as false on nulls.+* The grouped moments fast path no longer drops a user function inside `sum`.+* `medianMaybe` prints as `medianMaybe`, not `meanMaybe`.+* Parquet row-group pruning recognises `F.and`/`.&&` conjunctions.+* The expression wire format decodes `logBase`, `stddev` and+  `countDistinct`; `(^^)` encodes as `powFrac`.+ ## 3.7.0.0 +* `dataframe-viz`: column names are escaped in Vega-Lite field references+  (`sepal.length` → `sepal\.length`), so a chart of a dotted column no longer+  renders empty. Data keys are unchanged. * `dataframe-parquet`: a requested column is matched by its full name first,   so a top-level column named with a dot (`sepal.length`) is read and projected   as written; the last segment is only an alias for nested leaves. Fixes typed

@@ -35,25 +35,7 @@  # DataFrame -Tabular data analysis and symbolic AI in Haskell. Supports eager computation and lazy/deferred computation. API flexible enough to accomodate beginners to both data analysis or Haskell, and advanced type-safe pipeline.--## Features--**I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from Hugging Face datasets (`hf://` URIs) via the `dataframe-huggingface` package. Column projection and predicate pushdown for Parquet reads.--**Operations**: filter, select, derive, groupBy, aggregate, joins (inner, left, right, full outer), sort, sample, stratified sample, distinct, k-fold splits.--**Expressions**: typed column references (`F.col @Double "x"`), arithmetic, comparisons, logical operators, nullable-aware three-valued logic (`.==`, `.&&`), string matching (`like`, `regex`), casting, and user-defined functions via `lift`/`lift2`.--**Statistics**: mean, median, mode, variance, standard deviation, percentiles, inter-quartile range, correlation, skewness, frequency tables, imputation.--**Plotting**: terminal plots (histogram, scatter, line, bar, box, pie, heatmap, stacked bar, correlation matrix) and interactive HTML plots.--**Lazy engine**: streaming query execution for files that don't fit in memory. Rule-based optimizer with filter fusion, predicate pushdown, and dead column elimination. Pull-based executor with configurable batch sizes.--**Interop**: Arrow C Data Interface for zero-copy round-trips with Python and Polars.--**ML**: decision trees (TAO algorithm), feature synthesis, k-fold cross-validation, stratified sampling.+Tabular data analysis and symbolic AI in Haskell. Supports eager computation and lazy/deferred computation. API flexible enough to accommodate beginners to both data analysis or Haskell, and advanced type-safe pipelines.  **Notebooks**: IHaskell integration with [pre-built Binder examples](https://mybinder.org/v2/gh/mchav/ihaskell-dataframe/HEAD). @@ -63,11 +45,13 @@  * Concise, declarative, composable data pipelines and machine learning. * Choose your level of type safety: keep it lightweight for quick analysis, or lock it down for production pipelines.-* High performance from Haskell's optimizing compiler and an efficient columnar memory model based onn Apache Arrow.+* High performance from Haskell's optimizing compiler and an efficient columnar memory model based on Apache Arrow. * Designed for interactivity: a custom REPL, IHaskell/Sabela notebook support, terminal and web plotting, and helpful error messages.  ## Install +The easiest way to get started is to install the standalone REPL.+ ```bash cabal update cabal install dataframe@@ -97,7 +81,7 @@ import qualified DataFrame.Functions as F import DataFrame.Expression.Operators -import DataFrame ((=:))+import DataFrame ((=:), as)  sales = D.fromColumns     [ "product" =: [1, 1, 2, 2, 3, 3 :: Int]@@ -107,8 +91,8 @@ -- Group by product and compute totals sales     |> D.groupBy ["product"]-    |> D.aggregate [ F.sum (F.col @Int "amount") `as` "total"-                   , F.count (F.col @Int "amount") `as` "orders"+    |> D.aggregate [ F.sum (F.int "amount") `as` "total"+                   , F.count (F.int "amount") `as` "orders"                    ]     |> D.toMarkdown' ```@@ -166,7 +150,7 @@ > | latitude            | 20640                    | 0                    | Double       | > | longitude           | 20640                    | 0                    | Double       | -The `:declareColumns` macro (`$(D.declareColumns df)` outside the REPL) generates typed column references from a dataframe, so you can use column names directly in expressions instead of writing `F.col @Double "median_income"` every time:+The `:declareColumns` macro (`$(D.declareColumns df)` outside the REPL) generates typed column references from a dataframe, so you can use column names directly in expressions instead of writing `F.double "median_income"` every time:  ```haskell $(D.declareColumns df)@@ -216,10 +200,6 @@         '(latitude + ocean_proximity)' ``` -## Template Haskell--For scripts and projects, Template Haskell can generate column bindings at compile time.- ### Generate column references from a CSV  `declareColumnsFromCsvFile` (in `DataFrame.TH`, also re-exported from `DataFrame`)@@ -235,7 +215,7 @@  df <- D.readCsv "./data/housing.csv" df |> D.derive "rooms_per_household" (total_rooms / households)-   |> D.filterWhere (median_income .>. 5)+   |> D.filter (median_income .>. 5)    |> D.groupBy ["ocean_proximity"]    |> D.aggregate [F.mean median_house_value `as` "avg_value"]    |> D.toMarkdown'@@ -254,8 +234,8 @@ ```haskell -- Without TH — every column needs its name and type spelled out df |> D.derive "rooms_per_household"-        (F.col @Double "total_rooms" / F.col @Double "households")-   |> D.filterWhere (F.col @Double "median_income" .>. F.lit 5)+        (F.double "total_rooms" / F.double "households")+   |> D.filter (F.double "median_income" .>. F.lit 5)    |> D.take 5    |> D.toMarkdown' ```@@ -336,7 +316,7 @@ And the typed bridge — `[Order]` to `TypedDataFrame OrderSchema` and back:  ```haskell-DT.thaw (DT.fromRecordsTyped xs :: DT.TypedDataFrame OrderSchema) |> D.toMarkdown'+DT.thaw (DT.fromRecords xs :: DT.TypedDataFrame OrderSchema) |> D.toMarkdown' ```  > <!-- scripths:mime text/plain -->@@ -368,7 +348,7 @@ orders :: IO D.DataFrame orders = do     raw <- D.readCsvWithSchema orderSchema "./data/orders.csv"-    pure (D.filter orderAmount (> 100) raw)+    pure (D.filterBy (> 100) orderAmount raw) ```  > <!-- scripths:mime text/plain -->@@ -417,7 +397,7 @@     Nothing  -> "Schema mismatch!"     Just tdf -> tdf         |> DT.derive @"bonus" (DT.col @"salary" * DT.lit 0.1)-        |> DT.filterWhere (DT.col @"salary" DT..>. DT.lit 50000)+        |> DT.filter (DT.col @"salary" DT..>. DT.lit 50000)         |> DT.select @'["name", "bonus"]         |> DT.thaw         |> D.toMarkdown'@@ -435,11 +415,11 @@  ```text -- Typo in column name -> compile error-tdf |> DT.filterWhere (DT.col @"slary" DT..>. DT.lit 50000)+tdf |> DT.filter (DT.col @"slary" DT..>. DT.lit 50000) -- error: Column 'slary' not found in schema  -- Wrong type -> compile error-tdf |> DT.filterWhere (DT.col @"name" DT..>. DT.lit 50000)+tdf |> DT.filter (DT.col @"name" DT..>. DT.lit 50000) -- error: Couldn't match type 'Text' with 'Double' ``` @@ -468,7 +448,7 @@  ## Lazy Queries -For files too large to fit in memory, `DataFrame.Lazy` provides a streaming query engine. Declare a schema, build a query plan with the same familiar operations, and `runDataFrame` runs it through an optimizer before streaming results batch-by-batch:+For files too large to fit in memory, `DataFrame.Lazy` provides a streaming query engine. Declare a schema, build a query plan with the same operations as the eager API, and `run` passes it through an optimizer before streaming results batch-by-batch:  ```haskell import qualified DataFrame.Lazy as L@@ -487,11 +467,11 @@     , ("ocean_proximity",    schemaType @Text)     ] -lazyResult <- L.runDataFrame $+lazyResult <- L.run $     L.scanCsv housingSchema "./data/housing.csv"-    |> L.filter  (F.col @Double "median_income" .>. F.lit 5)+    |> L.filter  (F.double "median_income" .>. F.lit 5)     |> L.derive  "value_per_income"-                 (F.col @Double "median_house_value" / F.col @Double "median_income")+                 (F.double "median_house_value" / F.double "median_income")     |> L.select  ["ocean_proximity", "median_house_value", "value_per_income"]     |> L.take 1000 @@ -513,6 +493,23 @@ > | NEAR BAY                | 373600.0                     | 58895.860264211624         |  The optimizer pushes the filter into the scan, drops unreferenced columns before reading, and stops pulling batches once 1000 rows have been collected.++### Switching between APIs++The eager, typed, lazy and typed-lazy APIs share operation names and argument+order, so moving a pipeline between them is mostly a change of imports:++| | Frame ops (`D`) | Expressions (`F`) |+|---|---|---|+| Eager | `DataFrame` | `DataFrame.Functions` |+| Typed | `DataFrame.Typed` | `DataFrame.Typed.Functions` |+| Lazy | `DataFrame.Lazy` | `DataFrame.Functions` |+| Typed lazy | `DataFrame.Typed.Lazy` | `DataFrame.Typed.Functions` |++A lazy pipeline starts from `scanCsv`/`scanParquet`/`fromDataFrame` and ends with+`run`; ops that return a value (`mean`, `nRows`, `writeCsv`, …) run the query+and return `IO`. Typed APIs name columns at the type level (`col @"x"`,+`select @'["x"]`) where untyped ones take text.  ## Documentation 

@@ -1,6 +1,5 @@ {-# LANGUAGE NumericUnderscores #-} {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  import Data.Time import qualified Data.Vector.Unboxed as VU@@ -23,14 +22,14 @@     let generationTime = diffUTCTime endGeneration startGeneration     putStrLn $ "Data generation Time: " ++ show generationTime     startCalculation <- getCurrentTime-    print $ D.mean (F.col @Double "0") df-    print $ D.variance (F.col @Double "1") df+    print $ D.mean (F.double "0") df+    print $ D.variance (F.double "1") df     print $ D.correlation "1" "2" df     endCalculation <- getCurrentTime     let calculationTime = diffUTCTime endCalculation startCalculation     putStrLn $ "Calculation Time: " ++ show calculationTime     startFilter <- getCurrentTime-    print $ D.filter (F.col @Double "0") (> 0.971) df D.|> D.take 10+    print $ D.filterBy (> 0.971) (F.double "0") df D.|> D.take 10     endFilter <- getCurrentTime     let filterTime = diffUTCTime endFilter startFilter     putStrLn $ "Filter Time: " ++ show filterTime

@@ -186,18 +186,18 @@                     ]      runQuery "Q1 — preview first 20 rows (no filter)" $-        L.runDataFrame $+        L.run $             L.take 20 $                 L.scanCsv schema pathT      runQuery "Q2 — filter (x > 0.999), limit 20" $-        L.runDataFrame $+        L.run $             L.take 20 $                 L.filter (col @Double "x" .> lit (0.999 :: Double)) $                     L.scanCsv schema pathT      runQuery "Q3 — filter (x > 0.999), derive z = x*y, select [id,z], limit 20" $-        L.runDataFrame $+        L.run $             L.take 20 $                 L.select ["id", "z"] $                     L.derive "z" (col @Double "x" * col @Double "y") $@@ -205,7 +205,7 @@                             L.scanCsv schema pathT      runQuery "Q4 — filter fusion: (x > 0.5) . (y > 0.5), limit 20" $-        L.runDataFrame $+        L.run $             L.take 20 $                 L.filter (col @Double "y" .> lit (0.5 :: Double)) $                     L.filter (col @Double "x" .> lit (0.5 :: Double)) $@@ -216,7 +216,7 @@             ++ approx (n `div` 1000)             ++ " rows expected)"         )-        $ L.runDataFrame+        $ L.run         $ L.select ["id", "x"]         $ L.filter (col @Double "x" .> lit (0.999 :: Double))         $ L.scanCsv schema pathT

@@ -13,6 +13,7 @@ import DataFrame.Expression.Operators import qualified DataFrame.Functions as F import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF import System.Random  $( DT.deriveSchemaFromCsvFileWith@@ -31,7 +32,7 @@     '[ '("Survived", Maybe Int), '("prediction", Int)]  prediction :: D.Expr Int-prediction = F.col @Int "prediction"+prediction = F.int "prediction"  main :: IO () main = do@@ -66,7 +67,7 @@                             }                     }                 )-                (F.fromMaybe 0 (F.col @(Maybe Int) "Survived"))+                (F.fromMaybe 0 (F.maybeInt "Survived"))                 (trainDf |> D.exclude ["PassengerId"])         model = predict fitted @@ -106,9 +107,9 @@         ) clean tdf =     tdf-        |> DT.replaceColumn @"Ticket" (DT.nullLift (T.filter isAlpha) (DT.col @"Ticket"))-        |> DT.replaceColumn @"Name" (DT.nullLift extractTitle (DT.col @"Name"))-        |> DT.replaceColumn @"Cabin" (DT.nullLift (T.take 1) (DT.col @"Cabin"))+        |> DT.replaceColumn @"Ticket" (DTF.nullLift (T.filter isAlpha) (DT.col @"Ticket"))+        |> DT.replaceColumn @"Name" (DTF.nullLift extractTitle (DT.col @"Name"))+        |> DT.replaceColumn @"Cabin" (DTF.nullLift (T.take 1) (DT.col @"Cabin"))         |> DT.renameMany             @'[ '("Name", "title")               , '("Cabin", "cabin_prefix")@@ -133,7 +134,7 @@                     df |> D.select ["Survived", "prediction"]         survived = DT.col @"Survived"         predCol = DT.col @"prediction"-        count expr = fromIntegral (DT.nRows (DT.filterWhere expr tdf))+        count expr = fromIntegral (DT.nRows (DT.filter expr tdf))         tp = count ((survived DT..==. DT.lit 1) DT..&&. (predCol DT..==. DT.lit 1))         tn = count ((survived DT..==. DT.lit 0) DT..&&. (predCol DT..==. DT.lit 0))         fp = count ((survived DT..==. DT.lit 0) DT..&&. (predCol DT..==. DT.lit 1))

@@ -1,6 +1,5 @@ {-# LANGUAGE NumericUnderscores #-} {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-} {-# OPTIONS_GHC -fno-warn-orphans #-}  import qualified DataFrame as D@@ -55,9 +54,9 @@         df             |> D.groupBy ["ocean_proximity"]             |> D.aggregate-                [ F.minimum (F.col @Double "median_house_value")+                [ F.minimum (F.double "median_house_value")                     `as` "minimum_median_house_value"-                , F.maximum (F.col @Double "median_house_value")+                , F.maximum (F.double "median_house_value")                     `as` "maximum_median_house_value"                 ] 

@@ -1,6 +1,6 @@ cabal-version:      3.4 name:               dataframe-version:            3.7.0.0+version:            3.8.0.0 synopsis: A fast, safe, and intuitive DataFrame library.  description: A fast, safe, and intuitive DataFrame library for exploratory data analysis.@@ -75,7 +75,8 @@     -- gets added to build-depends but never re-exported.     ghc-options: -Werror=unused-packages     exposed-modules: DataFrame,-                    DataFrame.Typed+                    DataFrame.Typed,+                    DataFrame.Typed.Interop     reexported-modules: DataFrame.Functions,                         DataFrame.Synthesis,                         DataFrame.Display.Web.Plot,@@ -114,6 +115,7 @@                         DataFrame.Boosting.AdaBoost,                         DataFrame.Metrics,                         DataFrame.IO.JSON,+                        DataFrame.Typed.IO.JSON,                         DataFrame.Expr.Serialize,                         DataFrame.IR.ExprJson,                         DataFrame.Typed.Types,@@ -124,21 +126,23 @@                         DataFrame.Typed.Join,                         DataFrame.Typed.Aggregate,                         DataFrame.Typed.Expr,+                        DataFrame.Typed.Functions,                         DataFrame.Typed.Record,                         DataFrame.Typed.Generic     build-depends:    base >= 4 && <5,-                      dataframe-core >= 2.6 && < 2.7,-                      dataframe-json >= 1.2.0.1 && < 1.3,-                      dataframe-expr-serializer >= 1.3 && < 1.4,-                      dataframe-operations >= 2.6 && < 2.7,+                      dataframe-core >= 2.7 && < 2.8,+                      dataframe-json >= 1.2.1 && < 1.3,+                      dataframe-expr-serializer >= 1.4 && < 1.5,+                      dataframe-operations >= 2.7 && < 2.8,                       dataframe-parsing >= 2.2 && < 2.3,-                      dataframe-viz >= 1.3.2 && < 1.4,-                      dataframe-learn >= 2.5 && < 2.6+                      dataframe-viz >= 1.4 && < 1.5,+                      dataframe-learn >= 2.5 && < 2.6,+                      text >= 2.1 && < 3      if !flag(no-csv)         reexported-modules: DataFrame.IO.CSV,                             DataFrame.Typed.IO.CSV-        build-depends:   dataframe-csv >= 2.3 && < 2.4+        build-depends:   dataframe-csv >= 2.3.2 && < 2.4         cpp-options:     -DWITH_CSV      if !flag(no-parquet)@@ -157,7 +161,7 @@                             DataFrame.IO.Parquet.Time,                             DataFrame.IO.Utils.RandomAccess,                             DataFrame.Typed.IO.Parquet-        build-depends:   dataframe-parquet >= 1.6 && < 1.7+        build-depends:   dataframe-parquet >= 1.6.1 && < 1.7         cpp-options:     -DWITH_PARQUET      -- The lazy executor calls both CSV and Parquet readers directly, so@@ -167,7 +171,7 @@                             DataFrame.Lazy.IO.Binary,                             DataFrame.Lazy.IO.CSV,                             DataFrame.Typed.Lazy-        build-depends:   dataframe-lazy >= 2.4.1 && < 2.5+        build-depends:   dataframe-lazy >= 2.5 && < 2.6         cpp-options:     -DWITH_LAZY      if !flag(no-th)@@ -191,8 +195,8 @@     import: warnings     main-is: Benchmark.hs     build-depends:    base >= 4 && < 5,-                      dataframe >= 3.7 && < 3.8,-                      dataframe-operations >= 2.6 && < 2.7,+                      dataframe >= 3.8 && < 3.9,+                      dataframe-operations >= 2.7 && < 2.8,                       random >= 1 && < 2,                       time >= 1.12 && < 2,                       vector >= 0.13 && < 0.15,@@ -204,10 +208,10 @@     import: warnings     main-is: Synthesis.hs     build-depends:    base >= 4 && < 5,-                      dataframe >= 3.7 && < 3.8,-                      dataframe-core >= 2.6 && < 2.7,+                      dataframe >= 3.8 && < 3.9,+                      dataframe-core >= 2.7 && < 2.8,                       dataframe-learn >= 2.5 && < 2.6,-                      dataframe-operations >= 2.6 && < 2.7,+                      dataframe-operations >= 2.7 && < 2.8,                       random >= 1 && < 2,                       text >= 2.1 && < 3     hs-source-dirs:   app@@ -238,9 +242,9 @@     build-depends:    base >= 4 && < 5,                       bytestring >= 0.11 && < 0.14,                       containers >= 0.6.7 && < 0.10,-                      dataframe >= 3.7 && < 3.8,-                      dataframe-core >= 2.6 && < 2.7,-                      dataframe-lazy >= 2.4.1 && < 2.5,+                      dataframe >= 3.8 && < 3.9,+                      dataframe-core >= 2.7 && < 2.8,+                      dataframe-lazy >= 2.5 && < 2.6,                       dataframe-parsing >= 2.2 && < 2.3,                       directory >= 1.3.0.0 && < 2,                       random >= 1 && < 2,@@ -259,9 +263,9 @@                    criterion >= 1 && < 2,                    deepseq >= 1.4 && < 2,                    process >= 1.6 && < 2,-                   dataframe >= 3.7 && < 3.8,-                   dataframe-core >= 2.6 && < 2.7,-                   dataframe-operations >= 2.6 && < 2.7,+                   dataframe >= 3.8 && < 3.9,+                   dataframe-core >= 2.7 && < 2.8,+                   dataframe-operations >= 2.7 && < 2.8,                    random >= 1 && < 2,     default-language: Haskell2010     ghc-options:@@ -319,6 +323,7 @@                    Operations.Inference,                    Operations.InsertColumn,                    Operations.Join,+                   Operations.OperatorIdentity,                    Operations.Merge,                    Operations.Nullable,                    Operations.NullableHashing,@@ -340,6 +345,7 @@                    Operations.Record,                    LazyParquet,                    LazyParity,+                   Parity.FourWay,                    LazyProjection,                    Parquet,                    ParquetTestData,@@ -353,18 +359,19 @@     build-depends:  base >= 4 && < 5,                     aeson >= 0.11.0.0 && < 3,                     bytestring >= 0.11 && < 0.14,-                    dataframe >= 3.7 && < 3.8,-                    dataframe-core >= 2.6 && < 2.7,-                    dataframe-core >= 2.6 && < 2.7,-                    dataframe-csv >= 2.3 && < 2.4,-                    dataframe-expr-serializer >= 1.3 && < 1.4,+                    dataframe >= 3.8 && < 3.9,+                    dataframe-core >= 2.7 && < 2.8,+                    dataframe-core >= 2.7 && < 2.8,+                    dataframe-csv >= 2.3.2 && < 2.4,+                    dataframe-expr-serializer >= 1.4 && < 1.5,                     dataframe-fastcsv >= 1.4.0.1 && < 1.5,-                    dataframe-json >= 1.2.0.1 && < 1.3,-                    dataframe-lazy >= 2.4.1 && < 2.5,+                    dataframe-json >= 1.2.1 && < 1.3,+                    dataframe-lazy >= 2.5 && < 2.6,                     dataframe-learn >= 2.5 && < 2.6,-                    dataframe-operations >= 2.6 && < 2.7,-                    dataframe-parquet >= 1.6 && < 1.7,+                    dataframe-operations >= 2.7 && < 2.8,+                    dataframe-parquet >= 1.6.1 && < 1.7,                     dataframe-parsing >= 2.2 && < 2.3,+                    dataframe-th >= 2.3 && < 2.4,                     HUnit >= 1.6 && < 1.8,                     QuickCheck >= 2 && < 3,                     random-shuffle >= 0.0.4 && < 1,@@ -389,9 +396,9 @@     other-modules: Internal.PackedText     build-depends:  base >= 4 && < 5,                     bytestring >= 0.11 && < 0.14,-                    dataframe >= 3.7 && < 3.8,-                    dataframe-core >= 2.6 && < 2.7,-                    dataframe-operations >= 2.6 && < 2.7,+                    dataframe >= 3.8 && < 3.9,+                    dataframe-core >= 2.7 && < 2.8,+                    dataframe-operations >= 2.7 && < 2.8,                     HUnit >= 1.6 && < 1.8,                     text >= 2.1 && < 3,                     vector >= 0.13 && < 0.15

@@ -173,7 +173,7 @@ import DataFrame.Lazy as Lazy (     LazyDataFrame,     fromDataFrame,-    runDataFrame,+    run,     scanCsv,     scanCsvWith,     scanParquet,@@ -192,6 +192,7 @@     fullOuterJoin,     innerJoin,     join,+    joinOn,     leftJoin,     rightJoin,  )@@ -243,7 +244,6 @@     filterBy,     filterJust,     filterNothing,-    filterWhere,     kFolds,     randomSplit,     range,

@@ -8,17 +8,13 @@     TypedGrouped,     These (..), -    -- * Typed expressions+    -- * Typed expressions (combinators live in "DataFrame.Typed.Functions")     TExpr (..),     col,     lit,     ifThenElse,     caseWhen,     orElse,-    lift,-    lift2,-    nullLift,-    nullLift2,      -- * Same-type comparison operators     (.==.),@@ -28,11 +24,21 @@     (.>=.),     (.>.), +    -- * Same-type arithmetic operators+    (.+.),+    (.-.),+    (.*.),+    (./.),+    (.^^.),+    (.^.),+     -- * Nullable-aware arithmetic operators     (.+),     (.-),     (.*),     (./),+    (.^^),+    (.^),      -- * Nullable-aware comparison operators (three-valued logic)     (.==),@@ -45,66 +51,11 @@     -- * Logical operators     (.&&.),     (.||.),-    DataFrame.Typed.Expr.not,--    -- * Aggregation expression combinators-    DataFrame.Typed.Expr.sum,-    mean,-    median,-    count,-    countAll,-    DataFrame.Typed.Expr.minimum,-    DataFrame.Typed.Expr.maximum,-    collect,-    over,--    -- * Expression combinators (full DataFrame.Functions parity)-    DataFrame.Typed.Expr.div,-    DataFrame.Typed.Expr.mod,-    mode,-    sumMaybe,-    DataFrame.Typed.Expr.meanMaybe,-    DataFrame.Typed.Expr.variance,-    DataFrame.Typed.Expr.medianMaybe,-    DataFrame.Typed.Expr.percentile,-    stddev,-    stddevMaybe,-    zScore,-    pow,-    relu,-    DataFrame.Typed.Expr.min,-    DataFrame.Typed.Expr.max,-    reduce,-    toMaybe,-    fromMaybe,-    isJust,-    isNothing,-    fromJust,-    whenPresent,-    whenBothPresent,-    recode,-    recodeWithCondition,-    recodeWithDefault,-    firstOrNothing,-    lastOrNothing,-    splitOn,-    match,-    matchAll,-    parseDate,-    daysBetween,-    bind,--    -- * Cast / coercion expressions-    castExpr,-    castExprWithDefault,-    castExprEither,-    unsafeCastExpr,-    toDouble,+    (.&&),+    (.||),      -- * Typed sort orders     TSortOrder (..),-    asc,-    desc,      -- * Freeze / thaw boundary     freeze,@@ -124,7 +75,6 @@     toIntMatrix,      -- * Schema-preserving operations-    filterWhere,     filter,     filterBy,     filterAllJust,@@ -132,6 +82,7 @@     filterNothing,     filterAllNothing,     sortBy,+    topBy,     take,     takeLast,     drop,@@ -144,6 +95,7 @@      -- * Schema-modifying operations     derive,+    deriveMany,     impute,     select,     exclude,@@ -156,6 +108,12 @@     dropColumn,     replaceColumn, +    -- * Construction and folds+    empty,+    null,+    fold,+    (|>),+     -- * Metadata     dimensions,     nRows,@@ -172,6 +130,9 @@     symmetricDifference,      -- * Joins+    JoinType (..),+    join,+    joinOn,     innerJoin,     leftJoin,     rightJoin,@@ -184,7 +145,9 @@     aggregateUntyped,      -- * Column transformations-    applyColumn,+    apply,+    applyInt,+    applyDouble,     applyMany,     applyWhere,     applyAtIndex,@@ -206,6 +169,24 @@     valueCounts,     valueProportions, +    -- * Statistics+    mean,+    meanMaybe,+    median,+    medianMaybe,+    percentile,+    genericPercentile,+    standardDeviation,+    skewness,+    variance,+    interQuartileRange,+    sum,+    correlation,+    frequencies,+    imputeWith,+    summarize,+    describeColumns,+ #ifdef WITH_TH     -- * Template Haskell     deriveSchema,@@ -224,9 +205,49 @@      -- * Record bridge (ADT <-> TypedDataFrame)     HasSchema (..),-    fromRecordsTyped,-    toRecordsTyped,+    fromRecords,+    toRecords, +    -- * Display and rendering+    DisplayOptions (..),+    defaultDisplayOptions,+    display,+    prettyPrint,+    toMarkdown,+    toMarkdown',+    toCsv,+    toCsv',+    toSeparated,++    -- * IO+    readJSON,+    readJSONEither,+#ifdef WITH_CSV+    ReadOptions (..),+    defaultReadOptions,+    readCsv,+    readCsvWithError,+    readCsvWithOpts,+    readTsv,+    readSeparated,+    writeCsv,+    writeTsv,+    writeSeparated,+#endif+#ifdef WITH_PARQUET+    ParquetReadOptions (..),+    ParquetWriteOptions (..),+    defaultParquetReadOptions,+    defaultParquetWriteOptions,+    readParquet,+    readParquetWithError,+    readParquetWithOpts,+    readParquetFiles,+    readParquetFilesWithOpts,+    writeParquet,+    writeParquetWithOptions,+#endif+     -- * Generics opt-in for schema derivation     SchemaOf,     SchemaOfRaw,@@ -268,7 +289,7 @@     AllKnownSymbol (..), ) where -import Prelude hiding (drop, filter, take)+import Prelude hiding (drop, filter, null, sum, take)  import DataFrame.Typed.Access (     columnAsDoubleVector,@@ -288,8 +309,10 @@     groupBy,  ) import DataFrame.Typed.Apply (+    apply,+    applyDouble,+    applyInt,     applyAtIndex,-    applyColumn,     applyMany,     applyWhere,     deriveWithExpr,@@ -306,7 +329,60 @@     stratifiedSample,     stratifiedSplit,  )-import DataFrame.Typed.Expr+import DataFrame.Typed.Expr (+    TExpr (..),+    caseWhen,+    col,+    ifThenElse,+    lit,+    orElse,+    (.&&),+    (.&&.),+    (.*),+    (.*.),+    (.+),+    (.+.),+    (.-),+    (.-.),+    (./),+    (./.),+    (./=),+    (./=.),+    (.<),+    (.<.),+    (.<=),+    (.<=.),+    (.==),+    (.==.),+    (.>),+    (.>.),+    (.>=),+    (.>=.),+    (.^),+    (.^.),+    (.^^),+    (.^^.),+    (.||),+    (.||.),+ )+import DataFrame.Typed.Statistics (+    correlation,+    describeColumns,+    frequencies,+    genericPercentile,+    imputeWith,+    interQuartileRange,+    mean,+    meanMaybe,+    median,+    medianMaybe,+    percentile,+    skewness,+    standardDeviation,+    summarize,+    sum,+    variance,+ ) import DataFrame.Typed.Freeze (freeze, freezeWithError, thaw, unsafeFreeze) import DataFrame.Typed.Generic (     NameCase (..),@@ -315,13 +391,61 @@     genericFromColumns,     genericToColumns,  )-import DataFrame.Typed.Join (fullOuterJoin, innerJoin, leftJoin, rightJoin)-import DataFrame.Typed.Operations-import DataFrame.Typed.Record (-    HasSchema (..),-    fromRecordsTyped,-    toRecordsTyped,+import DataFrame.Display (DisplayOptions (..), defaultDisplayOptions)+import DataFrame.Expression.Operators ((|>))+import DataFrame.Operations.Join (JoinType (..))+#ifdef WITH_CSV+import DataFrame.IO.CSV (ReadOptions (..), defaultReadOptions)+import DataFrame.Typed.IO.CSV (+    readCsv,+    readCsvWithError,+    readCsvWithOpts,+    readSeparated,+    readTsv,+    writeCsv,+    writeSeparated,+    writeTsv,  )+#endif+#ifdef WITH_PARQUET+import DataFrame.IO.Parquet (+    ParquetReadOptions (..),+    ParquetWriteOptions (..),+    defaultParquetReadOptions,+    defaultParquetWriteOptions,+ )+import DataFrame.Typed.IO.Parquet (+    readParquet,+    readParquetFiles,+    readParquetFilesWithOpts,+    readParquetWithError,+    readParquetWithOpts,+    writeParquet,+    writeParquetWithOptions,+ )+#endif+import DataFrame.Typed.IO.JSON (readJSON, readJSONEither)+import DataFrame.Typed.Interop (+    display,+    fromRecords,+    prettyPrint,+    toCsv,+    toCsv',+    toMarkdown,+    toMarkdown',+    toRecords,+    toSeparated,+ )+import DataFrame.Typed.Join (+    fullOuterJoin,+    innerJoin,+    join,+    joinOn,+    leftJoin,+    rightJoin,+ )+import DataFrame.Typed.Operations+import DataFrame.Typed.Record (HasSchema (..)) import DataFrame.Typed.Schema #ifdef WITH_TH import DataFrame.Typed.TH (

@@ -0,0 +1,63 @@+{-# LANGUAGE ScopedTypeVariables #-}++-- | Typed display, rendering and record helpers mirroring "DataFrame".+module DataFrame.Typed.Interop (+    -- * Display and rendering+    display,+    prettyPrint,+    toMarkdown,+    toMarkdown',+    toCsv,+    toCsv',+    toSeparated,++    -- * Records+    fromRecords,+    toRecords,+) where++import qualified Data.Text as T++import qualified DataFrame.Core as Core+import DataFrame.Display (DisplayOptions)+import qualified DataFrame.Display as Display+import DataFrame.Typed.Freeze (thaw)+import DataFrame.Typed.Record (HasSchema (..), fromRecordsTyped, toRecordsTyped)+import DataFrame.Typed.Types (TExpr (..), TypedDataFrame)++-- | Print the frame with the given options.+display :: DisplayOptions -> TypedDataFrame cols -> IO ()+display opts = Display.display opts . thaw++-- | Render a typed expression as source-like text.+prettyPrint :: TExpr cols a -> String+prettyPrint (TExpr e) = Core.prettyPrint e++-- | Render the frame as a Markdown table.+toMarkdown :: TypedDataFrame cols -> T.Text+toMarkdown = Core.toMarkdown . thaw++-- | 'toMarkdown' as a 'String'.+toMarkdown' :: TypedDataFrame cols -> String+toMarkdown' = Core.toMarkdown' . thaw++-- | Render the frame as CSV.+toCsv :: TypedDataFrame cols -> T.Text+toCsv = Core.toCsv . thaw++-- | 'toCsv' as a 'String'.+toCsv' :: TypedDataFrame cols -> String+toCsv' = Core.toCsv' . thaw++-- | Render the frame with the given separator.+toSeparated :: Char -> TypedDataFrame cols -> T.Text+toSeparated sep = Core.toSeparated sep . thaw++-- | Build a typed frame from records; the schema comes from the record type.+fromRecords :: forall a. (HasSchema a) => [a] -> TypedDataFrame (Schema a)+fromRecords = fromRecordsTyped++-- | Parse records back out of a typed frame.+toRecords ::+    forall a. (HasSchema a) => TypedDataFrame (Schema a) -> Either T.Text [a]+toRecords = toRecordsTyped

@@ -5,11 +5,9 @@  import Data.Time.Calendar (Day, fromGregorian) import qualified DataFrame as D-import DataFrame.Functions (-    sanitize,- ) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI+import DataFrame.TH.Identifier (sanitize) import Test.HUnit  -- Test cases for the sanitize function@@ -77,7 +75,7 @@                 , ("sum", DI.fromList (replicate 10 (55 :: Int)))                 ]             )-            (D.derive "sum" (F.sum (F.col @Int "A")) df)+            (D.derive "sum" (F.sum (F.int "A")) df)         )  testPow :: Test@@ -88,7 +86,7 @@             [4, 9, 16, 25, 36, 49, 64, 81, 100, 121]             ( D.columnAsList @Int                 (F.col "sq")-                (D.derive "sq" (F.pow (F.col @Int "A" + F.lit 1) 2) df)+                (D.derive "sq" (F.pow (F.int "A" + F.lit 1) 2) df)             )         ) 

@@ -76,16 +76,17 @@ -- Row-wise expression node kinds. exprRoundtripTests :: [Test] exprRoundtripTests =-    [ testCase "col" (roundtrips (F.col @Double "x"))+    [ testCase "col" (roundtrips (F.double "x"))     , testCase "lit" (roundtrips (F.lit @Int 42))-    , testCase "arithmetic" (roundtrips (F.col @Double "x" * F.lit 2 + F.lit 1))-    , testCase "division" (roundtrips (F.col @Double "x" / F.lit 2))-    , testCase "comparison" (roundtrips (F.col @Double "x" .>. F.lit 0))-    , testCase "unary sqrt" (roundtrips (sqrt (F.col @Double "x")))+    , testCase "arithmetic" (roundtrips (F.double "x" * F.lit 2 + F.lit 1))+    , testCase "division" (roundtrips (F.double "x" / F.lit 2))+    , testCase "comparison" (roundtrips (F.double "x" .>. F.lit 0))+    , testCase "unary sqrt" (roundtrips (sqrt (F.double "x")))+    , testCase "logBase" (roundtrips (logBase (F.lit 2) (F.double "x")))     , testCase         "if/then/else"         ( roundtrips-            ( ifThenElse (F.col @Double "x" .>. F.lit 0) (F.lit 1.0) (F.lit (-1.0)) ::+            ( ifThenElse (F.double "x" .>. F.lit 0) (F.lit 1.0) (F.lit (-1.0)) ::                 Expr Double             )         )@@ -94,14 +95,16 @@ -- Aggregation and window node kinds (the decoder cases added for full parity). aggRoundtripTests :: [Test] aggRoundtripTests =-    [ testCase "sum" (roundtrips (F.sum (F.col @Double "x")))-    , testCase "count" (roundtrips (F.count (F.col @Double "x")))-    , testCase "minimum" (roundtrips (F.minimum (F.col @Double "x")))-    , testCase "maximum" (roundtrips (F.maximum (F.col @Int "x")))-    , testCase "mean" (roundtrips (F.mean (F.col @Double "x")))-    , testCase "variance" (roundtrips (F.variance (F.col @Double "x")))-    , testCase "median" (roundtrips (F.median (F.col @Double "x")))-    , testCase "over" (roundtrips (F.over ["g"] (F.sum (F.col @Double "x"))))+    [ testCase "sum" (roundtrips (F.sum (F.double "x")))+    , testCase "count" (roundtrips (F.count (F.double "x")))+    , testCase "minimum" (roundtrips (F.minimum (F.double "x")))+    , testCase "maximum" (roundtrips (F.maximum (F.int "x")))+    , testCase "mean" (roundtrips (F.mean (F.double "x")))+    , testCase "variance" (roundtrips (F.variance (F.double "x")))+    , testCase "median" (roundtrips (F.median (F.double "x")))+    , testCase "stddev" (roundtrips (F.stddev (F.double "x")))+    , testCase "countDistinct" (roundtrips (F.countDistinct (F.int "x")))+    , testCase "over" (roundtrips (F.over ["g"] (F.sum (F.double "x"))))     ]  -- Documented limitation: collect's list-typed output cannot be encoded.@@ -109,7 +112,10 @@ limitationTests =     [ testCase         "collect cannot encode"-        (isLeft (encodeExpr (F.collect (F.col @Double "x"))))+        (isLeft (encodeExpr (F.collect (F.double "x"))))+    , testCase+        "anonymous udf cannot encode"+        (isLeft (encodeExpr (F.lift (+ 1) (F.double "x"))))     ]  -- Pin the wire shape Python decodes against: (x * 2) + 1 over doubles.@@ -142,7 +148,7 @@ ioTests =     [ TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do         let fp = dir ++ "/expr.json"-            e = F.col @Double "x" * F.lit 2 + F.lit 1+            e = F.double "x" * F.lit 2 + F.lit 1         sr <- saveExprToFile fp e         assertEqual "save expr ok" (Right ()) sr         loaded <- loadExprAtFromFile @Double fp@@ -155,8 +161,8 @@     , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do         let fp = dir ++ "/pipeline.json"             nes =-                [ ("z", SomeExpr (F.col @Double "x" * F.lit 2))-                , ("w", SomeExpr (F.col @Double "x" + F.lit 10))+                [ ("z", SomeExpr (F.double "x" * F.lit 2))+                , ("w", SomeExpr (F.double "x" + F.lit 10))                 ]         sr <- savePipelineToFile fp nes         assertEqual "save pipeline ok" (Right ()) sr@@ -165,11 +171,11 @@             Left err -> assertFailure ("load pipeline failed: " <> err)             Right nes' -> do                 let df' = applyTransform (Transform nes') df3-                assertBool "z = 2x" (close (interpD df' (F.col @Double "z")) [2, 4, 6])-                assertBool "w = x+10" (close (interpD df' (F.col @Double "w")) [11, 12, 13])+                assertBool "z = 2x" (close (interpD df' (F.double "z")) [2, 4, 6])+                assertBool "w = x+10" (close (interpD df' (F.double "w")) [11, 12, 13])     , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do         let fp = dir ++ "/transform.json"-            t = Transform [("z", SomeExpr (F.col @Double "x" * F.lit 2))]+            t = Transform [("z", SomeExpr (F.double "x" * F.lit 2))]         sr <- saveTransformToFile fp t         assertEqual "save transform ok" (Right ()) sr         loaded <- loadTransformFromFile fp@@ -178,11 +184,11 @@             Right t' ->                 assertBool                     "transform z = 2x"-                    (close (interpD (applyTransform t' df3) (F.col @Double "z")) [2, 4, 6])+                    (close (interpD (applyTransform t' df3) (F.double "z")) [2, 4, 6])     , TestCase $ withSystemTempDirectory "expr-ser" $ \dir -> do         -- End-to-end: fit OLS, persist its prediction, reload, compare outputs.         let fp = dir ++ "/model.json"-            m = fit defaultLinearConfig (F.col @Double "y") regDF+            m = fit defaultLinearConfig (F.double "y") regDF         sr <- saveExprToFile fp (predict m)         assertEqual "save model ok" (Right ()) sr         loaded <- loadExprAtFromFile @Double fp

@@ -87,11 +87,11 @@  orderingParity :: Test orderingParity = TestCase $ do-    let sortedPacked = D.sortBy [D.Asc (F.col @T.Text "k")] packedDf-        sortedBoxed = D.sortBy [D.Asc (F.col @T.Text "k")] boxedDf+    let sortedPacked = D.sortBy [D.Asc (F.text "k")] packedDf+        sortedBoxed = D.sortBy [D.Asc (F.text "k")] boxedDf     assertBool "asc sort parity" (sortedPacked == sortedBoxed)-    let descPacked = D.sortBy [D.Desc (F.col @T.Text "k")] packedDf-        descBoxed = D.sortBy [D.Desc (F.col @T.Text "k")] boxedDf+    let descPacked = D.sortBy [D.Desc (F.text "k")] packedDf+        descBoxed = D.sortBy [D.Desc (F.text "k")] boxedDf     assertBool "desc sort parity" (descPacked == descBoxed)  hashingParity :: Test@@ -135,8 +135,8 @@ -- A sort on a packed column stays packed and equals the boxed sort. sortPreservesPacked :: Test sortPreservesPacked = TestCase $ do-    let sortedP = D.sortBy [D.Asc (F.col @T.Text "k")] packedDf-        sortedB = D.sortBy [D.Asc (F.col @T.Text "k")] boxedDf+    let sortedP = D.sortBy [D.Asc (F.text "k")] packedDf+        sortedB = D.sortBy [D.Asc (F.text "k")] boxedDf     assertBool         "sorted packed column stays PackedText"         (DI.isPackedText (unsafeGetColumn "k" sortedP))@@ -215,7 +215,7 @@ eqLiteralKernelParity :: Test eqLiteralKernelParity = TestCase $ do     let base = D.fromColumns [("k", packedFromBytes kernelRowBytes)]-        sorted = D.sortBy [D.Asc (F.col @T.Text "k")] base+        sorted = D.sortBy [D.Asc (F.text "k")] base         boxed df = D.fromColumns [("k", DI.materializePacked (unsafeGetColumn "k" df))]         lits = ["apple", "", "café", "日本語", "\xFFFD", "caf", "missing"] :: [T.Text]         result :: D.DataFrame -> D.Expr Bool -> [Bool]@@ -231,10 +231,10 @@         | (label, df) <- [("base", base), ("sorted", sorted)]         , lit <- lits         , e <--            [ F.col @T.Text "k" .==. F.lit lit-            , F.col @T.Text "k" ./=. F.lit lit-            , F.lit lit .==. F.col @T.Text "k"-            , F.lit lit ./=. F.col @T.Text "k"+            [ F.text "k" .==. F.lit lit+            , F.text "k" ./=. F.lit lit+            , F.lit lit .==. F.text "k"+            , F.lit lit ./=. F.text "k"             ]         ] 

@@ -8,12 +8,11 @@ import Data.Text (Text) import qualified Data.Text as T import qualified DataFrame as D-import DataFrame.Expression.Operators (as, (|>))+import DataFrame.Expression.Operators (as, (.=), (.>.), (|>)) import qualified DataFrame.Functions as F import qualified DataFrame.IO.CSV as Csv import qualified DataFrame.Internal.Column as DI import qualified DataFrame.Internal.Expression as E-import DataFrame.Lazy (SortOrder (Descending)) import qualified DataFrame.Lazy as L import qualified DataFrame.Operations.Aggregation as Agg import DataFrame.Operations.Join (JoinType (LEFT))@@ -83,8 +82,8 @@     TestCase $         withCsv (ordersFrame 20_000) $ \ordersPath ->             withCsv (customersFrame 257) $ \customersPath -> do-                let amount = F.col @Double "amount"-                    discount = F.col @Double "discount"+                let amount = F.double "amount"+                    discount = F.double "discount"                     aggs =                         [ F.sum (amount - discount) `as` "revenue"                         , F.count amount `as` "orders"@@ -99,10 +98,11 @@                 let customersQ = L.scanCsv customersSchema (T.pack customersPath)                 lazy <-                     L.scanCsv ordersSchema (T.pack ordersPath)-                        |> (\o -> L.join LEFT "customer_id" "customer_id" o customersQ)-                        |> L.groupBy ["region", "plan"] aggs-                        |> L.sortBy [("revenue", Descending)]-                        |> L.runDataFrame+                        |> (\o -> L.join LEFT ["customer_id"] o customersQ)+                        |> L.groupBy ["region", "plan"]+                        |> L.aggregate aggs+                        |> L.sortBy [L.Desc (E.Col @Double "revenue")]+                        |> L.run                 assertEqual                     "join pipeline lazy == eager"                     (show eager)@@ -113,7 +113,7 @@ groupByPipelineParity =     TestCase $         withCsv (ordersFrame 20_000) $ \ordersPath -> do-            let amount = F.col @Double "amount"+            let amount = F.double "amount"                 aggs =                     [ F.sum amount `as` "total"                     , F.count amount `as` "n"@@ -125,9 +125,10 @@                             Agg.groupBy ["customer_id"] ordersDf             lazy <-                 L.scanCsv ordersSchema (T.pack ordersPath)-                    |> L.groupBy ["customer_id"] aggs-                    |> L.sortBy [("total", Descending)]-                    |> L.runDataFrame+                    |> L.groupBy ["customer_id"]+                    |> L.aggregate aggs+                    |> L.sortBy [L.Desc (E.Col @Double "total")]+                    |> L.run             assertEqual                 "groupBy pipeline lazy == eager"                 (show eager)@@ -140,8 +141,8 @@             case DI.fromList [(0, 0) :: (Double, Int)] of                 DI.UnboxedColumn{} -> pure ()                 _ -> assertFailure "streaming mean tuple must stay unboxed"-            let valueD = F.col @Double "amount"-                valueI = F.col @Int "order_id"+            let valueD = F.double "amount"+                valueI = F.int "order_id"                 aggs =                     [ F.mean valueD `as` "mean_double"                     , F.mean valueI `as` "mean_int"@@ -150,9 +151,10 @@                     (L.scanCsvStreamingWith Csv.readSeparated ordersSchema (T.pack csvPath))                         { L.batchSize = 2                         }-                        |> L.groupBy ["customer_id"] aggs-                        |> L.sortBy [("customer_id", Descending)]-            actual <- L.runDataFrame query+                        |> L.groupBy ["customer_id"]+                        |> L.aggregate aggs+                        |> L.sortBy [L.Desc (E.Col @Int "customer_id")]+            actual <- L.run query             let expected =                     Perm.sortBy [Perm.Desc (E.Col @Int "customer_id")] $                         Agg.aggregate aggs $@@ -174,5 +176,112 @@             , ("discount", DI.fromList (replicate 12 (0 :: Double)))             ] +-- | Streaming (unbounded) probe side through the streaming hash join.+streamingJoinParity :: Test+streamingJoinParity =+    TestCase $+        withCsv (ordersFrame 5_000) $ \ordersPath ->+            withCsv (customersFrame 257) $ \customersPath -> do+                ordersDf <- Csv.readCsvWithSchema ordersSchema ordersPath+                customersDf <- Csv.readCsvWithSchema customersSchema customersPath+                let byOrder = Perm.sortBy [Perm.Asc (E.Col @Int "order_id")]+                    orders =+                        (L.scanCsvStreaming ordersSchema (T.pack ordersPath))+                            { L.batchSize = 512+                            }+                    renamedCustomers = D.rename "customer_id" "cid" customersDf+                    check lbl eager lazy = do+                        actual <- L.run lazy+                        assertEqual lbl (show (byOrder eager)) (show (byOrder actual))+                check+                    "streaming inner join"+                    (Join.innerJoin ["customer_id"] ordersDf customersDf)+                    (L.innerJoin ["customer_id"] orders (L.fromDataFrame customersDf))+                check+                    "streaming left join"+                    (Join.leftJoin ["customer_id"] ordersDf customersDf)+                    (L.leftJoin ["customer_id"] orders (L.fromDataFrame customersDf))+                check+                    "streaming joinOn"+                    (Join.joinOn LEFT [("customer_id", "cid")] ordersDf renamedCustomers)+                    ( L.joinOn+                        LEFT+                        [("customer_id", "cid")]+                        orders+                        (L.fromDataFrame renamedCustomers)+                    )++-- | New plan nodes under a scan, where pruning and pushdown are active.+optimizedNodesParity :: Test+optimizedNodesParity =+    TestCase $+        withCsv (ordersFrame 3_000) $ \ordersPath -> do+            df <- Csv.readCsvWithSchema ordersSchema ordersPath+            let scan = (L.scanCsv ordersSchema (T.pack ordersPath)){L.batchSize = 256}+                amount = F.double "amount"+                check lbl eager lazy = do+                    actual <- L.run lazy+                    assertEqual lbl (show eager) (show actual)+            check+                "rename, filter on new name, select"+                ( D.select ["amt", "order_id"] $+                    D.filter (F.double "amt" .>. 1000) $+                        D.rename "amount" "amt" df+                )+                ( L.select ["amt", "order_id"] $+                    L.filter (F.double "amt" .>. 1000) $+                        L.rename "amount" "amt" scan+                )+            check+                "filter above drop stays above"+                (D.filter (amount .>. 100) (D.drop 50 df))+                (L.filter (amount .>. 100) (L.drop 50 scan))+            check+                "filter above apply moves below it"+                ( D.select ["order_id", "discount"] $+                    D.filter (amount .>. 4000) $+                        D.apply (* (2 :: Double)) "discount" df+                )+                ( L.select ["order_id", "discount"] $+                    L.filter (amount .>. 4000) $+                        L.apply (* (2 :: Double)) "discount" scan+                )+            check+                "filter on applied column stays above"+                ( D.filter+                    (F.double "discount" .>. 2)+                    (D.apply (* (2 :: Double)) "discount" df)+                )+                ( L.filter+                    (F.double "discount" .>. 2)+                    (L.apply (* (2 :: Double)) "discount" scan)+                )+            check+                "exclude then select"+                (D.select ["order_id"] (D.exclude ["discount"] df))+                (L.select ["order_id"] (L.exclude ["discount"] scan))+            check+                "filter above takeLast stays above"+                (D.filter (amount .>. 10) (D.takeLast 100 df))+                (L.filter (amount .>. 10) (L.takeLast 100 scan))+            check+                "deriveMany then select a derived column"+                ( D.select ["x"] $+                    D.deriveMany ["x" .= (amount * 2), "y" .= amount] df+                )+                ( L.select ["x"] $+                    L.deriveMany ["x" .= (amount * 2), "y" .= amount] scan+                )+            check+                "append two scans"+                (df <> D.take 10 df)+                (scan <> L.take 10 scan)+ tests :: [Test]-tests = [joinPipelineParity, groupByPipelineParity, streamingMeanParity]+tests =+    [ optimizedNodesParity+    , joinPipelineParity+    , groupByPipelineParity+    , streamingMeanParity+    , streamingJoinParity+    ]

@@ -9,7 +9,6 @@ import Data.Time (UTCTime) import qualified DataFrame as D import qualified DataFrame.Functions as F-import DataFrame.Lazy (SortOrder (..)) import qualified DataFrame.Lazy as L import DataFrame.Schema (Schema (..), schemaType) import Test.HUnit@@ -41,7 +40,7 @@     TestCase         ( do             eager <- D.readParquet plainPath-            actual <- L.runDataFrame (L.scanParquet allTypesSchema (T.pack plainPath))+            actual <- L.run (L.scanParquet allTypesSchema (T.pack plainPath))             assertEqual "basicScan dimensions" (D.dimensions eager) (D.dimensions actual)         ) @@ -51,7 +50,7 @@     TestCase         ( do             actual <--                L.runDataFrame+                L.run                     (L.select ["id", "bool_col"] (L.scanParquet allTypesSchema (T.pack plainPath)))             assertEqual "columnProjection" (8, 2) (D.dimensions actual)         )@@ -62,9 +61,9 @@     TestCase         ( do             actual <--                L.runDataFrame+                L.run                     ( L.filter-                        (F.geq (F.col @Int32 "id") (F.lit 6))+                        (F.geq (F.int32 "id") (F.lit 6))                         (L.scanParquet allTypesSchema (T.pack plainPath))                     )             assertEqual "filterPushdown" (2, 11) (D.dimensions actual)@@ -76,11 +75,11 @@     TestCase         ( do             actual <--                L.runDataFrame+                L.run                     ( L.select                         ["id", "bool_col"]                         ( L.filter-                            (F.geq (F.col @Int32 "id") (F.lit 6))+                            (F.geq (F.int32 "id") (F.lit 6))                             (L.scanParquet allTypesSchema (T.pack plainPath))                         )                     )@@ -93,7 +92,7 @@     TestCase         ( do             actual <--                L.runDataFrame+                L.run                     ( L.scanParquet                         allTypesSchema                         "./tests/data/alltypes_plain*.parquet"@@ -109,20 +108,31 @@     TestCase         ( do             actual <--                L.runDataFrame+                L.run                     ( L.take                         3                         ( L.sortBy-                            [("id", Ascending)]+                            [L.Asc (L.int32 "id")]                             (L.scanParquet allTypesSchema (T.pack plainPath))                         )                     )             assertEqual "sortAndLimit rows" 3 (fst (D.dimensions actual))         ) +-- | A scan returns its columns in file order, like an eager read.+fileColumnOrder :: Test+fileColumnOrder =+    TestCase+        ( do+            actual <- L.run (L.scanParquet allTypesSchema (T.pack plainPath))+            eager <- D.readParquet plainPath+            assertEqual "column order" (D.columnNames eager) (D.columnNames actual)+        )+ tests :: [Test] tests =-    [ basicScan+    [ fileColumnOrder+    , basicScan     , columnProjection     , filterPushdown     , filterAndProject

@@ -40,29 +40,29 @@ twoCols =     Schema $ M.fromList [("id", schemaType @Int), ("dob", schemaType @Int)] --- | The scan keeps only the schema's columns, whatever their header position.+-- | The scan keeps only the schema's columns, in header order like eager. scanProjects :: Test scanProjects = TestLabel "scan_projects" $ TestCase $ withCsv ',' $ \csvPath -> do-    df <- L.runDataFrame (L.scanCsv twoCols (T.pack csvPath))+    df <- L.run (L.scanCsv twoCols (T.pack csvPath))     assertEqual "dims" (2, 2) (D.dimensions df)-    assertEqual "columns" ["dob", "id"] (D.columnNames df)+    assertEqual "columns" ["id", "dob"] (D.columnNames df)  -- | Same schema, same projection, through the SIMD reader. scanProjectsWithFastReader :: Test scanProjectsWithFastReader =     TestLabel "scan_projects_fast_reader" $ TestCase $ withCsv ',' $ \csvPath -> do         df <--            L.runDataFrame+            L.run                 (L.scanCsvWith Fast.fastReadCsvWithOpts twoCols (T.pack csvPath))         assertEqual "dims" (2, 2) (D.dimensions df)-        assertEqual "columns" ["dob", "id"] (D.columnNames df)+        assertEqual "columns" ["id", "dob"] (D.columnNames df)  -- | Both readers must agree on what a scan produces. readersAgree :: Test readersAgree = TestLabel "scan_readers_agree" $ TestCase $ withCsv ',' $ \csvPath -> do-    slow <- L.runDataFrame (L.scanCsv twoCols (T.pack csvPath))+    slow <- L.run (L.scanCsv twoCols (T.pack csvPath))     fast <--        L.runDataFrame (L.scanCsvWith Fast.fastReadCsvWithOpts twoCols (T.pack csvPath))+        L.run (L.scanCsvWith Fast.fastReadCsvWithOpts twoCols (T.pack csvPath))     assertEqual "same frame" slow fast  -- | scanSeparated must honour its separator, not always split on commas.@@ -70,9 +70,9 @@ scanHonoursSeparator =     TestLabel "scan_honours_separator" $ TestCase $ withCsv ';' $ \csvPath -> do         -- With the separator ignored, the schema's columns are not found.-        df <- L.runDataFrame (L.scanSeparated ';' twoCols (T.pack csvPath))+        df <- L.run (L.scanSeparated ';' twoCols (T.pack csvPath))         assertEqual "dims" (2, 2) (D.dimensions df)-        assertEqual "columns" ["dob", "id"] (D.columnNames df)+        assertEqual "columns" ["id", "dob"] (D.columnNames df)  -- | A schema naming a column the file lacks is an error, not a short frame. scanMissingColumn :: Test@@ -80,7 +80,7 @@     let bogus = Schema $ M.fromList [("nope", schemaType @Int)]     r <-         try-            (L.runDataFrame (L.scanCsv bogus (T.pack csvPath)) >>= evaluate . forceDataFrame)+            (L.run (L.scanCsv bogus (T.pack csvPath)) >>= evaluate . forceDataFrame)     case r of         Left (e :: SomeException) ->             assertBool

@@ -42,12 +42,12 @@     xs2 = [2, 1, 4, 3, 6, 5, 8, 7] :: [Double]  col :: D.DataFrame -> T.Text -> [Double]-col d n = interpD d (F.col @Double n)+col d n = interpD d (F.double n)  -- | @interpret (predict m)@ must equal @intercept + Σ coefⱼ·featureⱼ@. linearDenotation :: Test linearDenotation = TestCase $ do-    let m = fit defaultLinearConfig (F.col @Double "y") df+    let m = fit defaultLinearConfig (F.double "y") df         feats = V.toList (regFeatureNames m)         coefs = VU.toList (regCoef m)         cols = map (col df) feats@@ -64,7 +64,7 @@ kmeansDenotation :: Test kmeansDenotation = TestCase $ do     let feats = ["x1", "x2"]-        km = fit defaultKMeansConfig{kmK = 3, kmSeed = 1} (map (F.col @Double) feats) df+        km = fit defaultKMeansConfig{kmK = 3, kmSeed = 1} (map F.double feats) df         centers = map VU.toList (V.toList (kmCenters km))         rows = transposeL (map (col df) feats)         native = [nearest centers row | row <- rows]

@@ -61,10 +61,10 @@         gb =             fit                 defaultGBConfig{gbNEstimators = 60, gbMaxDepth = 2}-                (F.col @Double "y")+                (F.double "y")                 df         preds = interpD df (predict gb)-        truth = interpD df (F.col @Double "y")+        truth = interpD df (F.double "y")         err = sum (zipWith (\p t -> (p - t) ^ (2 :: Int)) preds truth) / 12     assertBool "GBM fits training data well" (err < 0.05)     assertBool "trainScore recorded" (VU.length (gbTrainScore gb) == 60)@@ -76,13 +76,13 @@                 [ ("x", DI.fromList ([1 .. 8] :: [Double]))                 , ("y", DI.fromList ([1 .. 8] :: [Double]))                 ]-        gb = fit defaultGBConfig{gbNEstimators = 10} (F.col @Double "y") df+        gb = fit defaultGBConfig{gbNEstimators = 10} (F.double "y") df     assertBool "stage 0 valid" (isJust (gbExprAtStage 0 gb))     assertBool "out of range stage rejected" (isNothing (gbExprAtStage 11 gb))  testAdaBoost :: Test testAdaBoost = TestCase $ do-    let m = fit defaultAdaBoostConfig (F.col @Int "label") clsDF+    let m = fit defaultAdaBoostConfig (F.int "label") clsDF         preds = interpI clsDF (predict m)     assertEqual "AdaBoost separates" [0, 0, 0, 0, 1, 1, 1, 1] preds @@ -91,7 +91,7 @@     let m =             fit                 defaultGMMConfig{gmmK = 2, gmmSeed = 1}-                [F.col @Double "a", F.col @Double "b"]+                [F.double "a", F.double "b"]                 blobs         assigns = interpI blobs (predict m)     assertBool "GMM converged" (gmmConverged m)@@ -101,7 +101,7 @@     let m2 =             fit                 defaultGMMConfig{gmmK = 2, gmmSeed = 1}-                [F.col @Double "a", F.col @Double "b"]+                [F.double "a", F.double "b"]                 blobs     assertEqual "GMM deterministic" (gmmMeans m) (gmmMeans m2)     assertBool "BIC finite" (not (isNaN (gmmBIC m)) && not (isInfinite (gmmBIC m)))@@ -113,13 +113,13 @@                 [ ("a", DI.fromList ([0, 0.1, 0.2, 5, 5.1, 5.2, 50] :: [Double]))                 , ("b", DI.fromList ([0, 0.1, 0.0, 5, 5.0, 5.1, 50] :: [Double]))                 ]-        m = fit (DBSCANConfig 1.0 2) [F.col @Double "a", F.col @Double "b"] df+        m = fit (DBSCANConfig 1.0 2) [F.double "a", F.double "b"] df     assertEqual "two clusters" 2 (dbNClusters m)     assertEqual "last point is noise" (-1) (VU.last (dbLabels m))     let surrogate =             dbscanSurrogateExpr                 D.defaultTreeConfig-                [F.col @Double "a", F.col @Double "b"]+                [F.double "a", F.double "b"]                 m                 df         preds = interpI df surrogate@@ -142,11 +142,11 @@             let mdl =                     fit                         (LinearConfig (Ridge alpha) defaultSolverConfig)-                        (F.col @Double "y")+                        (F.double "y")                         (train :: D.DataFrame)              in r2                     (VU.fromList (interpD test' (predict mdl)))-                    (VU.fromList (interpD test' (F.col @Double "y")))+                    (VU.fromList (interpD test' (F.double "y")))         res = gridSearch 4 7 [0.0, 1.0, 100.0] score df     assertBool "best score high" (gsBestScore res > 0.99)     assertEqual "all configs scored" 3 (length (gsAll res))@@ -179,10 +179,10 @@                     , gbLearningRate = 0.1                     , gbMaxDepth = 3                     }-                (F.col @Double "label")+                (F.double "label")                 sigmoidCurveDF         probs = interpD sigmoidCurveDF (gbProbaExpr m)-        truth = map trueP (interpD sigmoidCurveDF (F.col @Double "x"))+        truth = map trueP (interpD sigmoidCurveDF (F.double "x"))         err =             sum (zipWith (\p t -> abs (p - t)) probs truth)                 / fromIntegral (length probs)@@ -214,10 +214,10 @@ testGBMBaseScore :: Test testGBMBaseScore = TestCase $ do     let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2}-        ys = interpD gbmData (F.col @Double "y")+        ys = interpD gbmData (F.double "y")         mean = sum ys / fromIntegral (length ys)-        plain = fit cfg (F.col @Double "y") gbmData-        based = fit cfg{gbBaseScore = Just (F.lit mean)} (F.col @Double "y") gbmData+        plain = fit cfg (F.double "y") gbmData+        based = fit cfg{gbBaseScore = Just (F.lit mean)} (F.double "y") gbmData         gap =             maximum                 ( zipWith@@ -229,15 +229,15 @@         ("base-score model matches the default, gap " ++ show gap)         (gap < 1e-9)     -- A per-row base is part of the prediction: shifting it shifts every score.-    let shifted = fit cfg{gbBaseScore = Just (F.col @Double "z")} (F.col @Double "y") gbmData+    let shifted = fit cfg{gbBaseScore = Just (F.double "z")} (F.double "y") gbmData         noTrees =             fit-                cfg{gbNEstimators = 0, gbBaseScore = Just (F.col @Double "z")}-                (F.col @Double "y")+                cfg{gbNEstimators = 0, gbBaseScore = Just (F.double "z")}+                (F.double "y")                 gbmData     assertEqual         "zero trees predict the base"-        (interpD gbmData (F.col @Double "z"))+        (interpD gbmData (F.double "z"))         (interpD gbmData (predict noTrees))     assertBool         "base model still fits"@@ -248,7 +248,7 @@ testGBMSubsample = TestCase $ do     let cfg = defaultGBConfig{gbNEstimators = 20, gbMaxDepth = 2}         scores :: GBConfig -> [Double]-        scores c = interpD gbmData (predict (fit c (F.col @Double "y") gbmData))+        scores c = interpD gbmData (predict (fit c (F.double "y") gbmData))     assertEqual         "subsample 1 / colsample 1 reproduce the default"         (scores cfg)

@@ -52,7 +52,7 @@ baseDF = mkRegDF [("x1", baseX1), ("x2", baseX2)] baseY  fitBase :: D.DataFrame -> LinearRegressor-fitBase = fit defaultLinearConfig (F.col @Double "y")+fitBase = fit defaultLinearConfig (F.double "y")  -- | Duplicating every row leaves the OLS coefficients unchanged. testDuplicateRows :: Test@@ -205,7 +205,7 @@              in (mu, sqrt var)     mapM_         ( \c -> do-            let (mu, sd) = moments (interpD scaledDF (F.col @Double c))+            let (mu, sd) = moments (interpD scaledDF (F.double c))             assertBool ("scaler: " ++ T.unpack c ++ " mean ~ 0") (close 1e-9 mu 0)             assertBool ("scaler: " ++ T.unpack c ++ " std ~ 1") (close 1e-9 sd 1)         )@@ -265,8 +265,8 @@ testR2OfFittedModel :: Test testR2OfFittedModel = TestCase $ do     let m = fitBase baseDF-        score = evaluate r2 (predict m) (F.col @Double "y") baseDF-        err = evaluate rmse (predict m) (F.col @Double "y") baseDF+        score = evaluate r2 (predict m) (F.double "y") baseDF+        err = evaluate rmse (predict m) (F.double "y") baseDF     assertBool "r2 of exact linear fit ~ 1" (close 1e-9 score 1.0)     assertBool "rmse of exact linear fit ~ 0" (err < 1e-6) @@ -289,13 +289,13 @@                 ("y" `notElem` getColumns expr)     assertNoTarget         "linear"-        (predict (fit defaultLinearConfig (F.col @Double "y") leakDF))+        (predict (fit defaultLinearConfig (F.double "y") leakDF))     assertNoTarget         "regression tree"-        (predict (fit defaultRegTreeConfig (F.col @Double "y") leakDF))+        (predict (fit defaultRegTreeConfig (F.double "y") leakDF))     assertNoTarget         "classification tree"-        (predict (fit D.defaultTreeConfig (F.col @Double "y") leakDF))+        (predict (fit D.defaultTreeConfig (F.double "y") leakDF))  tests :: [Test] tests =

@@ -1,6 +1,5 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ScopedTypeVariables #-}-{-# LANGUAGE TypeApplications #-}  module Learn.MetricsTests (tests) where @@ -91,10 +90,10 @@  testEvaluateOneLiner :: Test testEvaluateOneLiner = TestCase $ do-    let m = fit defaultLinearConfig (F.col @Double "y") reg-        score = evaluate rmse (predict m) (F.col @Double "y") reg+    let m = fit defaultLinearConfig (F.double "y") reg+        score = evaluate rmse (predict m) (F.double "y") reg     assertBool "evaluate rmse ~ 0 on exact linear fit" (score < 1e-6)-    let r = regressionReportExpr (predict m) (F.col @Double "y") reg+    let r = regressionReportExpr (predict m) (F.double "y") reg     assertBool "report r2 ~ 1" (close 1e-6 (rrR2 r) 1)  testCrossValidate :: Test@@ -104,8 +103,8 @@                 4                 0                 r2-                (F.col @Double "y")-                (predict . fit defaultLinearConfig (F.col @Double "y"))+                (F.double "y")+                (predict . fit defaultLinearConfig (F.double "y"))                 reg     assertBool "all folds high R2" (all (> 0.99) cv)     assertEqual "four folds" 4 (length cv)@@ -113,7 +112,7 @@ testTransformCompose :: Test testTransformCompose = TestCase $ do     let scaler = standardScaler ["x"] reg-        pca = fit (PCAConfig (NComp 1) False) [F.col @Double "x"] reg+        pca = fit (PCAConfig (NComp 1) False) [F.double "x"] reg         pipeline = scalerTransform scaler <> pcaTransform pca         out = applyTransform pipeline reg     assertBool "pipeline produced a frame" (D.columnNames out /= [])

@@ -72,13 +72,13 @@  testOLS :: Test testOLS = TestCase $ do-    let m = fit defaultLinearConfig (F.col @Double "y") regDF+    let m = fit defaultLinearConfig (F.double "y") regDF     assertBool         "OLS coef ~ [2,-3]"         (and (zipWith (close 1e-6) (VU.toList (regCoef m)) [2, -3]))     assertBool "OLS intercept ~ 1" (close 1e-6 (regIntercept m) 1)     let preds = interpD regDF (predict m)-        truth = interpD regDF (F.col @Double "y")+        truth = interpD regDF (F.double "y")     assertBool "OLS expr matches y" (and (zipWith (close 1e-6) preds truth))  -- | Regression frame whose feature x is stored as Int.@@ -92,7 +92,7 @@ -- | A non-Double feature column is a fit-time error, not a silent coercion. testFitRequiresDouble :: Test testFitRequiresDouble = TestCase $ do-    let m = fit defaultLinearConfig (F.col @Double "y") intFeatureDF+    let m = fit defaultLinearConfig (F.double "y") intFeatureDF     result <-         try (evaluate (VU.sum (regCoef m))) :: IO (Either DataFrameException Double)     case result of@@ -113,7 +113,7 @@ -- | Fitting rejects a nullable input column with an actionable error. testFitRejectsNullable :: Test testFitRejectsNullable = TestCase $ do-    let m = fit defaultLinearConfig (F.col @Double "y") nullableFeatureDF+    let m = fit defaultLinearConfig (F.double "y") nullableFeatureDF     result <-         try (evaluate (VU.sum (regCoef m))) :: IO (Either DataFrameException Double)     case result of@@ -128,14 +128,14 @@ testRidgeShrinks :: Test testRidgeShrinks = TestCase $ do     let r0 =-            fit (LinearConfig (Ridge 0.01) defaultSolverConfig) (F.col @Double "y") regDF-        r1 = fit (LinearConfig (Ridge 100) defaultSolverConfig) (F.col @Double "y") regDF+            fit (LinearConfig (Ridge 0.01) defaultSolverConfig) (F.double "y") regDF+        r1 = fit (LinearConfig (Ridge 100) defaultSolverConfig) (F.double "y") regDF         norm = VU.sum . VU.map abs . regCoef     assertBool "stronger ridge shrinks coefficients" (norm r1 < norm r0)  testLogistic :: Test testLogistic = TestCase $ do-    let m = fit defaultLogisticConfig (F.col @Int "label") clsDF+    let m = fit defaultLogisticConfig (F.int "label") clsDF         dec = predict m         preds = interpI clsDF dec         truth = [0, 0, 0, 0, 1, 1, 1, 1]@@ -155,22 +155,22 @@  testSVC :: Test testSVC = TestCase $ do-    let m = fit defaultSVCConfig (F.col @Int "label") clsDF+    let m = fit defaultSVCConfig (F.int "label") clsDF         preds = interpI clsDF (predict m)     assertEqual "linear SVC separates" [0, 0, 0, 0, 1, 1, 1, 1] preds  testRegressionTree :: Test testRegressionTree = TestCase $ do-    let m = fit defaultRegTreeConfig (F.col @Double "y") regDF+    let m = fit defaultRegTreeConfig (F.double "y") regDF         preds = interpD regDF (predict m)-        truth = interpD regDF (F.col @Double "y")+        truth = interpD regDF (F.double "y")         sse = sum (zipWith (\p t -> (p - t) ^ (2 :: Int)) preds truth)     assertBool "regression tree reduces error" (sse < 200)     assertBool "tree has leaves" (dtrNLeaves m >= 2)  testClassifierStats :: Test testClassifierStats = TestCase $ do-    let m = fit D.defaultTreeConfig (F.col @Int "label") clsDF+    let m = fit D.defaultTreeConfig (F.int "label") clsDF     assertBool "classifier depth >= 1" (dtcDepth m >= 1)  testPCA :: Test@@ -178,7 +178,7 @@     let m =             fit                 (PCAConfig (NComp 2) False)-                [F.col @Double "x1", F.col @Double "x2"]+                [F.double "x1", F.double "x2"]                 regDF         ratio = VU.toList (pcaExplainedVarianceRatio m)     assertBool "explained ratio sums ~1" (close 1e-9 (sum ratio) 1)@@ -199,11 +199,11 @@                 , ("b", DI.fromList ([0, 0.1, 0, 10, 10, 10.1] :: [Double]))                 ]         cfg = defaultKMeansConfig{kmK = 2, kmNInit = 5, kmSeed = 1}-        m = fit cfg [F.col @Double "a", F.col @Double "b"] df+        m = fit cfg [F.double "a", F.double "b"] df         labels = VU.toList (kmLabels m)     assertBool "two blobs split" (VU.head (kmLabels m) /= VU.last (kmLabels m))     assertEqual "two centers" 2 (V.length (kmCenters m))-    let m2 = fit cfg [F.col @Double "a", F.col @Double "b"] df+    let m2 = fit cfg [F.double "a", F.double "b"] df     assertEqual "kmeans deterministic" (kmCenters m) (kmCenters m2)     let assigns = interpI df (predict m)     assertEqual "assign expr matches labels" labels assigns@@ -213,7 +213,7 @@     let scaler = standardScaler ["x1", "x2"] regDF         t = scalerTransform scaler         scaledDf = applyTransform t regDF-        m = fit defaultLinearConfig (F.col @Double "y") scaledDf+        m = fit defaultLinearConfig (F.double "y") scaledDf         composed = compileThrough t (predict m)         viaCompose = interpD regDF composed         viaStepwise = interpD scaledDf (predict m)

@@ -67,7 +67,7 @@ recoveryTests :: [Test] recoveryTests =     [ "per-segment OLS recovered" ~: do-        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") twoSegDF+        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") twoSegDF             a = segModel (segByKey ["a"] m)             b = segModel (segByKey ["b"] m)         assertBool "two segments" (length (smSegments m) == 2)@@ -76,16 +76,16 @@         assertBool "b slope ~ -3" (close 1e-9 (regCoef b VU.! 0) (-3))         assertBool "b intercept ~ 5" (close 1e-9 (regIntercept b) 5)     , "prediction reproduces y" ~: do-        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") twoSegDF+        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") twoSegDF             preds = interpD twoSegDF (predict m)-            truth = interpD twoSegDF (F.col @Double "y")+            truth = interpD twoSegDF (F.double "y")         assertBool "routed predictions match y" (and (zipWith (close 1e-6) preds truth))     ]  routingTests :: [Test] routingTests =     [ "unseen category routes to fallback" ~: do-        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") twoSegDF+        let m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") twoSegDF             fb = smFallback m             unseen =                 D.fromColumns@@ -110,12 +110,12 @@                     , ("x", DI.fromList ([1, 2, 3, 4, 1, 2, 3, 4] :: [Double]))                     , ("label", DI.fromList labels)                     ]-            m = fit (lowFloor (segmented defaultLogisticConfig)) (F.col @T.Text "label") df+            m = fit (lowFloor (segmented defaultLogisticConfig)) (F.text "label") df             preds = interpT df (predict m)         assertBool "logistic predicts one label per row" (length preds == 8)     , "symbolic base composes" ~: do         let cfg = defaultSRConfig{srGenerations = 2, srPopSize = 12}-            m = fit (lowFloor (segmented cfg)) (F.col @Double "y") twoSegDF+            m = fit (lowFloor (segmented cfg)) (F.double "y") twoSegDF             preds = interpD twoSegDF (predict m)         assertBool "symbolic predicts one value per row" (length preds == 8)     ]@@ -133,7 +133,7 @@             m =                 fit                     (segmentOn (lowFloor (segmented defaultLinearConfig)) ["g"])-                    (F.col @Double "y")+                    (F.double "y")                     df         assertEqual "only g" ["g"] (smCatCols m)     , "cardinality cap skips high-cardinality text" ~: do@@ -145,7 +145,7 @@                     , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double]))                     ]             cfg = (lowFloor (segmented defaultLinearConfig)){segMaxCard = 2}-            m = fit cfg (F.col @Double "y") df+            m = fit cfg (F.double "y") df         assertEqual "k (8 distinct) skipped, g kept" ["g"] (smCatCols m)     ]   where@@ -163,7 +163,7 @@                     , ("x", DI.fromList ([1, 2, 3, 9, 1, 2, 3, 9] :: [Double]))                     , ("y", DI.fromList ([3, 5, 7, 0, 2, -1, -4, 0] :: [Double]))                     ]-            m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") df+            m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") df         -- Only "a" and "b" appear as segments; "null" is never a key.         assertBool             "no null-keyed segment"@@ -176,7 +176,7 @@     D.DataFrame -> IO (Either SomeException (SegmentedModel Double LinearRegressor)) fitErr df =     try-        (evaluate (fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") df))+        (evaluate (fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") df))  guardTests :: [Test] guardTests =@@ -221,7 +221,7 @@                     , ("n", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double]))                     , ("y", DI.fromList ([1, 2, 3, 4, 5, 6, 7, 8] :: [Double]))                     ]-            m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") df+            m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") df         assertEqual "two segments" 2 (length (smSegments m))     ] @@ -234,7 +234,7 @@                     , ("x", DI.fromList ([1, 2, 3, 4, 9] :: [Double]))                     , ("y", DI.fromList ([3, 5, 7, 9, 0] :: [Double]))                     ]-            m = fit (lowFloor (segmented defaultLinearConfig)) (F.col @Double "y") df+            m = fit (lowFloor (segmented defaultLinearConfig)) (F.double "y") df         assertEqual "one local segment" 1 (length (smSegments m))         assertEqual "segment a has 4 rows" 4 (segN (segByKey ["a"] m))         assertEqual "b fell back with 1 row" [(["b"], 1)] (smFellBack m)@@ -248,9 +248,9 @@                     2                     0                     rmse-                    (F.col @Double "y")+                    (F.double "y")                     ( predict-                        . fit ((segmented defaultLinearConfig){segMinRows = 2}) (F.col @Double "y")+                        . fit ((segmented defaultLinearConfig){segMinRows = 2}) (F.double "y")                     )                     twoSegDF         assertBool "two fold scores" (length scores == 2)@@ -265,7 +265,7 @@         let fitL lam =                 fit                     (pooled (lowFloor (segmented defaultLinearConfig)) lam)-                    (F.col @Double "y")+                    (F.double "y")                     twoSegDF             m0 = fitL 0             mInf = fitL 1e9@@ -292,7 +292,7 @@                 ( evaluate                     ( fit                         (pooled (lowFloor (segmented defaultSRConfig)) 1.0)-                        (F.col @Double "y")+                        (F.double "y")                         twoSegDF                     )                 ) ::

@@ -59,7 +59,7 @@     Left err -> error (show err)  colDoubles :: D.DataFrame -> T.Text -> [Double]-colDoubles df name = interpD df (F.col @Double name)+colDoubles df name = interpD df (F.double name)  closeAll :: Double -> [Double] -> [Double] -> Bool closeAll tol a b =@@ -74,7 +74,7 @@ testOLSParity = TestCase $ do     g <- loadGolden     df <- D.readCsv "data/ml/regression.csv"-    let m = fit defaultLinearConfig (F.col @Double "target") df+    let m = fit defaultLinearConfig (F.double "target") df         goldCoef = asNums (g .! "ols" .! "coef")         goldB = asNum (g .! "ols" .! "intercept")     assertBool@@ -87,7 +87,7 @@     g <- loadGolden     df <- D.readCsv "data/ml/regression.csv"     let m =-            fit (LinearConfig (Ridge 1.0) defaultSolverConfig) (F.col @Double "target") df+            fit (LinearConfig (Ridge 1.0) defaultSolverConfig) (F.double "target") df         goldCoef = asNums (g .! "ridge" .! "coef")     assertBool         "Ridge coefficients match sklearn"@@ -99,7 +99,7 @@     df <- D.readCsv "data/ml/iris.csv"     let feats =             map-                (F.col @Double)+                F.double                 ["sepal_length", "sepal_width", "petal_length", "petal_width"]         m = fit (PCAConfig (NComp 2) False) feats df         goldEvr = asNums (g .! "pca" .! "evr")@@ -116,7 +116,7 @@ testLogisticIrisParity = TestCase $ do     g <- loadGolden     df <- D.readCsv "data/ml/iris.csv"-    let m = fit defaultLogisticConfig (F.col @Double "species") df+    let m = fit defaultLogisticConfig (F.double "species") df         preds = interpD df (predict m)         truth = colDoubles df "species"         acc = accuracyOf preds truth@@ -129,7 +129,7 @@ testSVCParity = TestCase $ do     g <- loadGolden     df <- D.readCsv "data/ml/iris_binary.csv"-    let m = fit defaultSVCConfig (F.col @Double "label") df+    let m = fit defaultSVCConfig (F.double "label") df         preds = interpD df (predict m)         truth = colDoubles df "label"         acc = accuracyOf preds truth@@ -145,7 +145,7 @@     let m =             fit                 defaultGBConfig{gbLoss = LogisticDeviance, gbNEstimators = 100}-                (F.col @Double "label")+                (F.double "label")                 df         probs = interpD df (gbProbaExpr m)         preds = map (\p -> if p > 0.5 then 1 else 0) probs@@ -163,7 +163,7 @@     let m =             fit                 defaultKMeansConfig{kmK = 3, kmNInit = 10, kmSeed = 0}-                [F.col @Double "x", F.col @Double "y"]+                [F.double "x", F.double "y"]                 df         gold = asNum (g .! "kmeans" .! "inertia")     assertBool

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  -- | The synthesis enumerator recovers small exact features deterministically. module Learn.Synthesis (tests) where@@ -31,7 +30,7 @@ -- | Pearson r²: enumeration should find x·x and score it ~1. recoversQuadratic :: Test recoversQuadratic = TestCase $ do-    let sf = fit defaultSynthesisConfig (D.col @Double "y") quad+    let sf = fit defaultSynthesisConfig (D.double "y") quad     assertBool         ("quadratic r2 = " ++ show (sfScore sf))         (sfScore sf > 0.999 && sfScore sf <= 1.0001)@@ -40,13 +39,13 @@ exactRecoveryMSE :: Test exactRecoveryMSE = TestCase $ do     let sf =-            fit defaultSynthesisConfig{synLoss = MeanSquaredError} (D.col @Double "y") quad+            fit defaultSynthesisConfig{synLoss = MeanSquaredError} (D.double "y") quad     assertBool ("exact -mse = " ++ show (sfScore sf)) (sfScore sf > -1.0e-6)  -- | Division is enumerated (with the denominator guard): a/b is recovered. recoversRatio :: Test recoversRatio = TestCase $ do-    let sf = fit defaultSynthesisConfig (D.col @Double "y") ratio+    let sf = fit defaultSynthesisConfig (D.double "y") ratio     assertBool         ("ratio r2 = " ++ show (sfScore sf))         (sfScore sf > 0.999 && sfScore sf <= 1.0001)@@ -54,7 +53,7 @@ -- | The bank is non-trivial and its ranked features are distinct expressions. distinctFeatures :: Test distinctFeatures = TestCase $ do-    let sf = fit defaultSynthesisConfig (D.col @Double "y") quad+    let sf = fit defaultSynthesisConfig (D.double "y") quad         names = [D.prettyPrint e | (e, _) <- sfFeatures sf]     assertBool "synthesizes more than one feature" (length names > 1)     assertBool "ranked features are distinct" (length names == length (dedup names))@@ -64,8 +63,8 @@ -- | Same config and data give the same best expression. deterministic :: Test deterministic = TestCase $ do-    let a = fit defaultSynthesisConfig (D.col @Double "y") quad-        b = fit defaultSynthesisConfig (D.col @Double "y") quad+    let a = fit defaultSynthesisConfig (D.double "y") quad+        b = fit defaultSynthesisConfig (D.double "y") quad     assertEqual         "same best expression"         (D.prettyPrint (sfExpr a))@@ -79,7 +78,7 @@             "[Error Case]"             "synMaxAllocBytes"             ( print-                (D.prettyPrint (sfExpr (fit defaultSynthesisConfig (D.col @Double "y") wide)))+                (D.prettyPrint (sfExpr (fit defaultSynthesisConfig (D.double "y") wide)))             )         ) @@ -87,7 +86,7 @@ acceptsSmallSearch :: Test acceptsSmallSearch = TestCase $ do     let cfg = defaultSynthesisConfig{synMaxSize = 3}-        m = fit cfg (D.col @Double "y") wide+        m = fit cfg (D.double "y") wide     assertBool "a size-3 search over the wide frame returns" (sfScore m >= -1.0)  -- | 12 features over 3000 rows: the shape that killed the kernel.

@@ -65,7 +65,7 @@ testTypedFitParity :: Test testTypedFitParity = TestCase $ do     let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF-        onUntyped = fit defaultLinearConfig (F.col @Double "y") regDF+        onUntyped = fit defaultLinearConfig (F.double "y") regDF     assertBool         "fittedModel of typed fit == untyped fit"         (sameModel (fittedModel onTyped) onUntyped)@@ -74,10 +74,10 @@ testTypedPredict :: Test testTypedPredict = TestCase $ do     let onTyped = fit defaultLinearConfig (DT.col @"y") housesTDF-        onUntyped = fit defaultLinearConfig (F.col @Double "y") regDF+        onUntyped = fit defaultLinearConfig (F.double "y") regDF         typedExpr = DT.unTExpr (predict onTyped)         untypedExpr = predict onUntyped -- :: Expr Double-        truth = interpD regDF (F.col @Double "y")+        truth = interpD regDF (F.double "y")     assertBool         "typed predict matches untyped predict"         ( and@@ -92,7 +92,7 @@ testTypedRidge = TestCase $ do     let cfg = defaultLinearConfig{lcPenalty = Ridge 0.01}         onTyped = fit cfg (DT.col @"y") housesTDF-        onUntyped = fit cfg (F.col @Double "y") regDF+        onUntyped = fit cfg (F.double "y") regDF     assertBool         "ridge: fittedModel of typed fit == untyped fit"         (sameModel (fittedModel onTyped) onUntyped)

@@ -47,6 +47,7 @@ import qualified Operations.MonomorphicLoops import qualified Operations.Nullable import qualified Operations.NullableHashing+import qualified Operations.OperatorIdentity import qualified Operations.ParallelGroupBy import qualified Operations.ParallelJoin import qualified Operations.Provenance@@ -67,6 +68,7 @@ import qualified Operations.Window import qualified Operations.WriteCsv import qualified PackedTextMigration+import qualified Parity.FourWay import qualified Parquet import qualified Plotting import qualified PrettyPrint@@ -109,6 +111,7 @@             ++ Operations.Inference.tests             ++ Operations.InsertColumn.tests             ++ Operations.Join.tests+            ++ Operations.OperatorIdentity.tests             ++ Operations.Merge.tests             ++ Operations.Nullable.tests             ++ Operations.NullableHashing.tests@@ -137,6 +140,7 @@             ++ Parquet.tests             ++ LazyParquet.tests             ++ LazyParity.tests+            ++ Parity.FourWay.tests             ++ LazyProjection.tests             ++ Plotting.tests             ++ Simplify.tests

@@ -72,8 +72,8 @@             "sortByM sorts ascending on A"             [1, 2, 3]             ( D.columnAsList @Int-                (F.col @Int "A")-                (execFrameM verbFixture (sortByM [Asc (F.col @Int "A")]))+                (F.int "A")+                (execFrameM verbFixture (sortByM [Asc (F.int "A")]))             )         ) 

@@ -9,6 +9,7 @@ import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF  import Data.Function import DataFrame.Expression.Operators@@ -39,8 +40,8 @@             )             ( testData                 & D.groupBy ["test1"]-                & D.aggregate [F.count (F.col @Int "test2") `as` "test2"]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.aggregate [F.count (F.int "test2") `as` "test2"]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -57,7 +58,7 @@             ( testData                 & D.groupBy ["test1"]                 & D.aggregate [F.countAll `as` "n"]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -82,8 +83,8 @@                          , '("test6", Integer)                          ]                 & DT.groupBy @'["test1"]-                & DT.aggregate (DT.as @"n" DT.countAll)-                & DT.sortBy [DT.asc (DT.col @"test1")]+                & DT.aggregate (DT.as @"n" DTF.countAll)+                & DT.sortBy [DTF.asc (DT.col @"test1")]                 & DT.thaw             )         )@@ -109,8 +110,8 @@                          , '("test6", Integer)                          ]                 & DT.groupBy @'["test1"]-                & DT.aggregate (DT.as @"test2_count" (DT.count (DT.col @"test2")))-                & DT.sortBy [DT.asc (DT.col @"test1")]+                & DT.aggregate (DT.as @"test2_count" (DTF.count (DT.col @"test2")))+                & DT.sortBy [DTF.asc (DT.col @"test1")]                 & DT.thaw             )         )@@ -127,8 +128,8 @@             )             ( testData                 & D.groupBy ["test1"]-                & D.aggregate [F.mean (F.col @Int "test2") `as` "test2"]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.aggregate [F.mean (F.int "test2") `as` "test2"]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -153,8 +154,8 @@                          , '("test6", Integer)                          ]                 & DT.groupBy @'["test1"]-                & DT.aggregate (DT.as @"test2_mean" (DT.mean (DT.col @"test2")))-                & DT.sortBy [DT.asc (DT.col @"test1")]+                & DT.aggregate (DT.as @"test2_mean" (DTF.mean (DT.col @"test2")))+                & DT.sortBy [DTF.asc (DT.col @"test1")]                 & DT.thaw             )         )@@ -172,9 +173,9 @@             ( testData                 & D.groupBy ["test1"]                 & D.aggregate-                    [ F.mean (F.lift (fromIntegral @Int @Double) (F.col @Int "test2")) `as` "test2"+                    [ F.mean (F.lift (fromIntegral @Int @Double) (F.int "test2")) `as` "test2"                     ]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -190,8 +191,8 @@             )             ( testData                 & D.groupBy ["test1"]-                & D.aggregate [F.mean (F.col @Int "test2" + F.col @Int "test2") `as` "test2"]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.aggregate [F.mean (F.int "test2" + F.int "test2") `as` "test2"]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -208,10 +209,10 @@             ( testData                 & D.groupBy ["test1"]                 & D.aggregate-                    [ F.maximum (F.lift (fromIntegral @Int @Double) (F.col @Int "test2"))+                    [ F.maximum (F.lift (fromIntegral @Int @Double) (F.int "test2"))                         `as` "test2"                     ]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -228,8 +229,8 @@             ( testData                 & D.groupBy ["test1"]                 & D.aggregate-                    [F.maximum (F.col @Int "test2" + F.col @Int "test2") `as` "test2"]-                & D.sortBy [D.Asc (F.col @Int "test1")]+                    [F.maximum (F.int "test2" + F.int "test2") `as` "test2"]+                & D.sortBy [D.Asc (F.int "test1")]             )         ) @@ -247,7 +248,7 @@                 & D.drop 12                 & D.groupBy ["test1"]                 & D.aggregate-                    [F.sum (F.col @Int "test2") `as` "sum(test2)"]+                    [F.sum (F.int "test2") `as` "sum(test2)"]             )         ) @@ -319,7 +320,7 @@             ( D.nRows                 ( optGroupByDf                     & D.groupBy ["key"]-                    & D.aggregate [F.count (F.col @Int "val") `as` "val"]+                    & D.aggregate [F.count (F.int "val") `as` "val"]                 )             )         )

@@ -240,7 +240,7 @@         ( assertEqual             "impute fills Nothing with the given value"             (Just $ DI.UnboxedColumn Nothing (VU.fromList [1 :: Int, 0, 3]))-            (DI.getColumn "opt" $ impute (F.col @(Maybe Int) "opt") 0 imputeData)+            (DI.getColumn "opt" $ impute (F.maybeInt "opt") 0 imputeData)         )  imputeColumnNotFound :: Test@@ -249,7 +249,7 @@         ( assertExpectException             "[Error Case]"             (DE.columnNotFound "missing" "impute" (D.columnNames imputeData))-            (print $ impute (F.col @(Maybe Int) "missing") 0 imputeData)+            (print $ impute (F.maybeInt "missing") 0 imputeData)         )  imputeOnNonOptional :: Test@@ -258,7 +258,7 @@         ( assertExpectException             "[Error Case]"             "impute"-            (print $ impute (F.col @(Maybe Int) "plain") 0 imputeData)+            (print $ impute (F.maybeInt "plain") 0 imputeData)         )  -- | Only a column reference can be imputed; a compound expression throws.

@@ -1,6 +1,5 @@ {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE ScopedTypeVariables #-}-{-# LANGUAGE TypeApplications #-}  module Operations.CountDistinct where @@ -171,8 +170,8 @@                 expected                 ( df                     & D.groupBy ["g"]-                    & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"]-                    & D.sortBy [D.Asc (F.col @Int "g")]+                    & D.aggregate [F.countDistinct (F.int "x") `as` "u"]+                    & D.sortBy [D.Asc (F.int "g")]                 )             ) @@ -196,8 +195,8 @@                 expected                 ( df                     & D.groupBy ["g"]-                    & D.aggregate [F.countDistinct (F.col @Int "x") `as` "u"]-                    & D.sortBy [D.Asc (F.col @Int32 "g")]+                    & D.aggregate [F.countDistinct (F.int "x") `as` "u"]+                    & D.sortBy [D.Asc (F.int32 "g")]                 )             ) 

@@ -12,6 +12,7 @@ import qualified DataFrame.Internal.Column as DI import qualified DataFrame.Internal.DataFrame as DI import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF  import Test.HUnit @@ -40,7 +41,7 @@                     "test4"                     ( F.lift2                         (++)-                        (F.lift show (F.col @Int "test1"))+                        (F.lift show (F.int "test1"))                         (F.lift (: ([] :: [Char])) (F.col @Char "test3"))                     )                     testData@@ -56,10 +57,10 @@             ( DT.columnAsList @"test4" $                 DT.derive                     @"test4"-                    ( DT.lift2+                    ( DTF.lift2                         (++)-                        (DT.lift show (DT.col @"test1"))-                        (DT.lift (: ([] :: [Char])) (DT.col @"test3"))+                        (DTF.lift show (DT.col @"test1"))+                        (DTF.lift (: ([] :: [Char])) (DT.col @"test3"))                     )                     ( either                         (error . show)

@@ -28,20 +28,20 @@ model =     F.lit 0.1         * F.ifThenElse-            (F.toDouble (F.col @Int "Age") .<=. F.lit 40.5)+            (F.toDouble (F.int "Age") .<=. F.lit 40.5)             ( F.ifThenElse-                (F.col @T.Text "Class" ./=. F.lit "Business")+                (F.text "Class" ./=. F.lit "Business")                 (F.lit (-0.859))                 (F.lit 0.31)             )             ( F.ifThenElse-                (F.toDouble (F.col @Int "Departure/Arrival time convenient") .<=. F.lit 2.5)+                (F.toDouble (F.int "Departure/Arrival time convenient") .<=. F.lit 2.5)                 (F.lit 1.2)-                (F.col @Double "Arrival Delay in Minutes" / F.lit 10)+                (F.double "Arrival Delay in Minutes" / F.lit 10)             )         + F.lit 0.1             * F.ifThenElse-                (F.col @T.Text "Class" ./=. F.lit "Eco")+                (F.text "Class" ./=. F.lit "Eco")                 (F.lit 0.5)                 (F.lit (-0.25)) @@ -50,7 +50,7 @@     either         (error . show)         id-        (D.columnAsUnboxedVector (F.col @Double "s") (D.derive "s" e frame))+        (D.columnAsUnboxedVector (F.double "s") (D.derive "s" e frame))  testShowRoundTrip :: Test testShowRoundTrip = TestCase $ case parseShownAs @Double (show model) of

@@ -5,6 +5,7 @@  import qualified Data.Text as T import qualified DataFrame as D+import DataFrame.Expression.Operators ((.==.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI @@ -32,8 +33,8 @@     TestCase         ( assertExpectException             "[Error Case]"-            (D.columnNotFound "test0" "filter" (D.columnNames testData))-            (print $ D.filter (F.col @Int "test0") even testData)+            (D.columnNotFound "test0" "" (D.columnNames testData))+            (print $ D.filter (F.int "test0" .==. 0) testData)         )  filterColumnWrongType :: Test@@ -42,7 +43,7 @@         ( assertExpectException             "[Error Case]"             (D.typeMismatchError (show $ typeRep @Integer) (show $ typeRep @Int))-            (print $ D.filter (F.col @Integer "test1") even testData)+            (print $ D.filter (F.integer "test1" .==. 0) testData)         )  filterByColumnDoesNotExist :: Test@@ -50,8 +51,8 @@     TestCase         ( assertExpectException             "[Error Case]"-            (D.columnNotFound "test0" "filter" (D.columnNames testData))-            (print $ D.filterBy even (F.col @Int "test0") testData)+            (D.columnNotFound "test0" "filterBy" (D.columnNames testData))+            (print $ D.filterBy even (F.int "test0") testData)         )  filterByColumnWrongType :: Test@@ -60,7 +61,7 @@         ( assertExpectException             "[Error Case]"             (D.typeMismatchError (show $ typeRep @Integer) (show $ typeRep @Int))-            (print $ D.filterBy even (F.col @Integer "test1") testData)+            (print $ D.filterBy even (F.integer "test1") testData)         )  filterColumnInexistentValues :: Test@@ -69,7 +70,7 @@         ( assertEqual             "Non existent filter value returns no rows"             (0, 8)-            (D.dimensions $ D.filter (F.col @Int "test1") (< 0) testData)+            (D.dimensions $ D.filterBy (< 0) (F.int "test1") testData)         )  filterColumnAllValues :: Test@@ -78,7 +79,7 @@         ( assertEqual             "Filters all columns"             (26, 8)-            (D.dimensions $ D.filter (F.col @Int "test1") (const True) testData)+            (D.dimensions $ D.filterBy (const True) (F.int "test1") testData)         )  filterJustWAI :: Test

@@ -161,24 +161,24 @@             assertEqual                 ("count over " ++ show n ++ " rows")                 [n]-                (take 1 (run (operandColumns xsI) (F.count (F.col @Int "x"))))+                (take 1 (run (operandColumns xsI) (F.count (F.int "x"))))         , TestCase $             assertEqual                 ("countDistinct of an expression over " ++ show n ++ " rows")                 [length (takeWhile (< n) [0 .. 6])]-                (take 1 (run (operandColumns xsI) (F.countDistinct (F.col @Int "x" + 1))))+                (take 1 (run (operandColumns xsI) (F.countDistinct (F.int "x" + 1))))         , TestCase $             assertBool                 ("mean of ints over " ++ show n ++ " rows")                 ( all                     (approx (sumD / fromIntegral n))-                    (run (operandColumns xsI) (F.mean (F.col @Int "x")))+                    (run (operandColumns xsI) (F.mean (F.int "x")))                 )         , TestCase $             assertEqual                 ("mean of doubles over " ++ show n ++ " rows")                 [sum (take n smallDoubles) / fromIntegral n]-                (take 1 (run (operandColumns xsD) (F.mean (F.col @Double "x"))))+                (take 1 (run (operandColumns xsD) (F.mean (F.double "x"))))         ]       where         -- Seven distinct ints with a large spread; the sum fits in a Double.

@@ -12,6 +12,7 @@ import DataFrame.Internal.Column.Types (These (..)) import DataFrame.Operations.Join import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF import Test.HUnit  df1 :: D.DataFrame@@ -54,7 +55,7 @@                 , ("B", D.fromList ["B0" :: Text, "B1", "B2"])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (innerJoin ["key"] df1 df2))+            (D.sortBy [D.Asc (F.text "key")] (innerJoin ["key"] df1 df2))         )  testLeftJoin :: Test@@ -68,7 +69,7 @@                 , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (leftJoin ["key"] df1 df2))+            (D.sortBy [D.Asc (F.text "key")] (leftJoin ["key"] df1 df2))         )  testRightJoin :: Test@@ -82,7 +83,7 @@                 , ("B", D.fromList ["B0" :: Text, "B1", "B2"])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (rightJoin ["key"] df1 df2))+            (D.sortBy [D.Asc (F.text "key")] (rightJoin ["key"] df1 df2))         )  tdf1 :: DT.TypedDataFrame ['("key", Text), '("A", Text)]@@ -102,7 +103,7 @@                 , ("B", D.fromList ["B0" :: Text, "B1", "B2"])                 ]             )-            (DT.thaw $ DT.sortBy [DT.asc (DT.col @"key")] (DT.innerJoin @'["key"] tdf1 tdf2))+            (DT.thaw $ DT.sortBy [DTF.asc (DT.col @"key")] (DT.innerJoin @'["key"] tdf1 tdf2))         )  testLeftJoinTyped :: Test@@ -116,7 +117,7 @@                 , ("B", D.fromList [Just "B0", Just "B1" :: Maybe Text, Just "B2"])                 ]             )-            (DT.thaw $ DT.sortBy [DT.asc (DT.col @"key")] (DT.leftJoin @'["key"] tdf1 tdf2))+            (DT.thaw $ DT.sortBy [DTF.asc (DT.col @"key")] (DT.leftJoin @'["key"] tdf1 tdf2))         )  -- A right-hand frame whose payload column is already optional.@@ -147,7 +148,7 @@                     )                 ]             )-            (DT.thaw $ DT.sortBy [DT.asc (DT.col @"key")] joined)+            (DT.thaw $ DT.sortBy [DTF.asc (DT.col @"key")] joined)         )   where     joined ::@@ -166,7 +167,7 @@                 , ("B", D.fromList ["B0" :: Text, "B1", "B2"])                 ]             )-            (DT.thaw $ DT.sortBy [DT.asc (DT.col @"key")] (DT.rightJoin @'["key"] tdf1 tdf2))+            (DT.thaw $ DT.sortBy [DTF.asc (DT.col @"key")] (DT.rightJoin @'["key"] tdf1 tdf2))         )  staffDf :: D.DataFrame@@ -213,7 +214,7 @@                     )                 ]             )-            (D.sortBy [D.Asc (F.col @Text "Name")] (fullOuterJoin ["Name"] studentDf staffDf))+            (D.sortBy [D.Asc (F.text "Name")] (fullOuterJoin ["Name"] studentDf staffDf))         )  testFullOuterJoinUnboxedKey :: Test@@ -234,7 +235,7 @@                 ]             )             ( D.sortBy-                [D.Asc (F.col @Int "customer_id")]+                [D.Asc (F.int "customer_id")]                 (fullOuterJoin ["customer_id"] ordersDf customersDf)             )         )@@ -281,7 +282,7 @@                 , ("Ronly", D.fromList [10 :: Int, 11])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (innerJoin ["key"] dfL dfR))+            (D.sortBy [D.Asc (F.text "key")] (innerJoin ["key"] dfL dfR))         )  testLeftJoinWithCollisions :: Test@@ -299,7 +300,7 @@                 , ("Ronly", D.fromList [Just 10 :: Maybe Int, Just 11, Nothing])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (leftJoin ["key"] dfL dfR))+            (D.sortBy [D.Asc (F.text "key")] (leftJoin ["key"] dfL dfR))         )  testRightJoinWithCollisions :: Test@@ -317,7 +318,7 @@                 , ("Lonly", D.fromList [Just "L0" :: Maybe Text, Just "L1", Nothing])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (rightJoin ["key"] dfL dfR))+            (D.sortBy [D.Asc (F.text "key")] (rightJoin ["key"] dfL dfR))         )  testOuterJoinWithCollisions :: Test@@ -340,7 +341,7 @@                 , ("Ronly", D.fromList [Just 10 :: Maybe Int, Just 11, Nothing, Just 13])                 ]             )-            (D.sortBy [D.Asc (F.col @Text "key")] (fullOuterJoin ["key"] dfL dfR))+            (D.sortBy [D.Asc (F.text "key")] (fullOuterJoin ["key"] dfL dfR))         )  testInnerJoinMissingKey :: Test

@@ -38,7 +38,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @Int "x" .+ F.col @(Maybe Int) "y")+                    (F.int "x" .+ F.maybeInt "y")                     testData             )         )@@ -53,7 +53,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @(Maybe Int) "y" .+ F.col @Int "x")+                    (F.maybeInt "y" .+ F.int "x")                     testData             )         )@@ -68,7 +68,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @Int "x" .+ F.col @Int "x")+                    (F.int "x" .+ F.int "x")                     testData             )         )@@ -83,7 +83,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @(Maybe Int) "y" .+ F.col @(Maybe Int) "y")+                    (F.maybeInt "y" .+ F.maybeInt "y")                     testData             )         )@@ -100,7 +100,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @Int "x" .- F.col @(Maybe Int) "y")+                    (F.int "x" .- F.maybeInt "y")                     testData             )         )@@ -117,7 +117,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @Int "x" .== F.col @(Maybe Int) "y" :: Expr (Maybe Bool))+                    (F.int "x" .== F.maybeInt "y" :: Expr (Maybe Bool))                     testData             )         )@@ -132,7 +132,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @Int "x" .== F.lit (1 :: Int) :: Expr Bool)+                    (F.int "x" .== F.lit (1 :: Int) :: Expr Bool)                     testData             )         )@@ -149,7 +149,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.nullLift negate (F.col @(Maybe Int) "y") :: Expr (Maybe Int))+                    (F.nullLift negate (F.maybeInt "y") :: Expr (Maybe Int))                     testData             )         )@@ -164,7 +164,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.nullLift negate (F.col @Int "x") :: Expr Int)+                    (F.nullLift negate (F.int "x") :: Expr Int)                     testData             )         )@@ -179,7 +179,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.nullLift2 (+) (F.col @Int "x") (F.col @(Maybe Int) "y") :: Expr (Maybe Int))+                    (F.nullLift2 (+) (F.int "x") (F.maybeInt "y") :: Expr (Maybe Int))                     testData             )         )@@ -194,7 +194,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.nullLift2 (+) (F.col @(Maybe Int) "y") (F.col @Int "x") :: Expr (Maybe Int))+                    (F.nullLift2 (+) (F.maybeInt "y") (F.int "x") :: Expr (Maybe Int))                     testData             )         )@@ -209,7 +209,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.nullLift2 (+) (F.col @Int "x") (F.col @Int "x") :: Expr Int)+                    (F.nullLift2 (+) (F.int "x") (F.int "x") :: Expr Int)                     testData             )         )@@ -235,7 +235,7 @@             "Int .+ Double = Double"             (Just $ DI.fromList [2.5, 4.5, 6.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .+ F.col @Double "d") crossData+                D.derive "result" (F.int "x" .+ F.double "d") crossData             )         ) @@ -247,7 +247,7 @@             "Double .+ Int = Double"             (Just $ DI.fromList [2.5, 4.5, 6.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Double "d" .+ F.col @Int "x") crossData+                D.derive "result" (F.double "d" .+ F.int "x") crossData             )         ) @@ -261,7 +261,7 @@                 DI.fromVector (V.fromList [Just 11.5, Nothing, Just 33.5 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @(Maybe Int) "y" .+ F.col @Double "d") crossData+                D.derive "result" (F.maybeInt "y" .+ F.double "d") crossData             )         ) @@ -275,7 +275,7 @@                 DI.fromVector (V.fromList [Just 11.5, Nothing, Just 33.5 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .+ F.col @(Maybe Double) "md") crossData+                D.derive "result" (F.int "x" .+ F.maybeDouble "md") crossData             )         ) @@ -291,7 +291,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @(Maybe Int) "y" .+ F.col @(Maybe Double) "md")+                    (F.maybeInt "y" .+ F.maybeDouble "md")                     crossData             )         )@@ -304,7 +304,7 @@             "Int .- Double = Double"             (Just $ DI.fromList [-0.5, -0.5, -0.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .- F.col @Double "d") crossData+                D.derive "result" (F.int "x" .- F.double "d") crossData             )         ) @@ -316,7 +316,7 @@             "Int .* Double = Double"             (Just $ DI.fromList [1.5, 5.0, 10.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .* F.col @Double "d") crossData+                D.derive "result" (F.int "x" .* F.double "d") crossData             )         ) @@ -328,7 +328,7 @@             "Double .- Int = Double"             (Just $ DI.fromList [0.5, 0.5, 0.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Double "d" .- F.col @Int "x") crossData+                D.derive "result" (F.double "d" .- F.int "x") crossData             )         ) @@ -342,7 +342,7 @@                 DI.fromVector (V.fromList [Just 8.5, Nothing, Just 26.5 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @(Maybe Int) "y" .- F.col @Double "d") crossData+                D.derive "result" (F.maybeInt "y" .- F.double "d") crossData             )         ) @@ -357,7 +357,7 @@                     (V.fromList [Just (-9.5), Nothing, Just (-27.5) :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .- F.col @(Maybe Double) "md") crossData+                D.derive "result" (F.int "x" .- F.maybeDouble "md") crossData             )         ) @@ -374,7 +374,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @(Maybe Int) "y" .- F.col @(Maybe Double) "md")+                    (F.maybeInt "y" .- F.maybeDouble "md")                     crossData             )         )@@ -387,7 +387,7 @@             "Double .* Int = Double"             (Just $ DI.fromList [1.5, 5.0, 10.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Double "d" .* F.col @Int "x") crossData+                D.derive "result" (F.double "d" .* F.int "x") crossData             )         ) @@ -401,7 +401,7 @@                 DI.fromVector (V.fromList [Just 15.0, Nothing, Just 105.0 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @(Maybe Int) "y" .* F.col @Double "d") crossData+                D.derive "result" (F.maybeInt "y" .* F.double "d") crossData             )         ) @@ -415,7 +415,7 @@                 DI.fromVector (V.fromList [Just 10.5, Nothing, Just 91.5 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" .* F.col @(Maybe Double) "md") crossData+                D.derive "result" (F.int "x" .* F.maybeDouble "md") crossData             )         ) @@ -431,7 +431,7 @@             ( DI.getColumn "result" $                 D.derive                     "result"-                    (F.col @(Maybe Int) "y" .* F.col @(Maybe Double) "md")+                    (F.maybeInt "y" .* F.maybeDouble "md")                     crossData             )         )@@ -457,7 +457,7 @@             "Int ./ Double = Double"             (Just $ DI.fromList [2.0, 2.0, 2.0 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" ./ F.col @Double "d") divData+                D.derive "result" (F.int "x" ./ F.double "d") divData             )         ) @@ -469,7 +469,7 @@             "Double ./ Int = Double"             (Just $ DI.fromList [0.5, 0.5, 0.5 :: Double])             ( DI.getColumn "result" $-                D.derive "result" (F.col @Double "d" ./ F.col @Int "x") divData+                D.derive "result" (F.double "d" ./ F.int "x") divData             )         ) @@ -483,7 +483,7 @@                 DI.fromVector (V.fromList [Just 4.0, Nothing, Just 2.0 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @(Maybe Int) "y" ./ F.col @Double "d") divData+                D.derive "result" (F.maybeInt "y" ./ F.double "d") divData             )         ) @@ -497,7 +497,7 @@                 DI.fromVector (V.fromList [Just 2.0, Nothing, Just 2.0 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @Int "x" ./ F.col @(Maybe Double) "md") divData+                D.derive "result" (F.int "x" ./ F.maybeDouble "md") divData             )         ) @@ -511,7 +511,7 @@                 DI.fromVector (V.fromList [Just 4.0, Nothing, Just 2.0 :: Maybe Double])             )             ( DI.getColumn "result" $-                D.derive "result" (F.col @(Maybe Int) "y" ./ F.col @(Maybe Double) "md") divData+                D.derive "result" (F.maybeInt "y" ./ F.maybeDouble "md") divData             )         ) @@ -737,8 +737,8 @@ sumGroupedNullable df =     df         & D.groupBy ["k"]-        & D.aggregate [F.sumMaybe (F.col @(Maybe Double) "v") `as` "s"]-        & D.sortBy [D.Asc (F.col @T.Text "k")]+        & D.aggregate [F.sumMaybe (F.maybeDouble "v") `as` "s"]+        & D.sortBy [D.Asc (F.text "k")]  expectedGroupSums :: [Double] -> D.DataFrame expectedGroupSums sums =

@@ -0,0 +1,258 @@+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | Operators are identified by tag, never by display name.+module Operations.OperatorIdentity (tests) where++import qualified Data.Text as T+import Data.Time (Day)+import qualified DataFrame as D+import DataFrame.Expression.Operators+import qualified DataFrame.Functions as F+import DataFrame.Internal.Column (Columnable)+import DataFrame.Internal.Expression (Expr, eqExpr, structuralKey)+import DataFrame.Internal.Expression.Simplify (simplify)++import Test.HUnit++xs :: D.DataFrame+xs = D.fromColumns [("x", D.fromList [1, 2, 3 :: Int])]++derived :: Expr Int -> [Int]+derived e = D.columnAsList (F.int "y") (D.derive "y" e xs)++weird :: T.Text -> Expr Int -> Expr Int -> Expr Int+weird opName = lift2Decorated (\a b -> a * 100 + b) opName Nothing True 6++udfNamedLikeBuiltin :: Test+udfNamedLikeBuiltin =+    TestLabel "udf named add keeps its own function" . TestCase $+        assertEqual "" [101, 202, 303] (derived (weird "add" x x))+  where+    x = F.int "x"++udfOtherName :: Test+udfOtherName =+    TestLabel "udf with fresh name" . TestCase $+        assertEqual "" [101, 202, 303] (derived (weird "myadd" x x))+  where+    x = F.int "x"++anonymousNotEqual :: Test+anonymousNotEqual =+    TestLabel "anonymous udfs are not equal" . TestCase $+        assertBool "" (not (eqExpr (F.lift2 (+) x x) (F.lift2 (*) x x)))+  where+    x = F.int "x"++namedEqual :: Test+namedEqual =+    TestLabel "same named op is equal" . TestCase $+        assertBool "" (eqExpr (weird "f" x x) (weird "f" x x))+  where+    x = F.int "x"++namedVsBuiltin :: Test+namedVsBuiltin =+    TestLabel "named add differs from builtin add" . TestCase $+        assertBool "" (not (eqExpr (weird "add" x x) (x + x)))+  where+    x = F.int "x"++fromMaybeDefaults :: Test+fromMaybeDefaults =+    TestLabel "fromMaybe defaults are part of identity" . TestCase $+        assertBool "" (not (eqExpr (F.fromMaybe 0 m) (F.fromMaybe 1 m)))+  where+    m = F.maybeInt "m"++powVariants :: Test+powVariants =+    TestLabel "^ and ^^ are different ops" . TestCase $+        assertBool "" (not (eqExpr (d .^. two) (d .^^. two)))+  where+    d = F.double "d"+    two = F.lit (2 :: Int)++rowsKept :: D.DataFrame -> Expr Bool -> Int+rowsKept df p = D.nRows (D.filter p df)++-- Both sides must keep the same rows before and after simplification.+sameRows :: String -> D.DataFrame -> Expr Bool -> Test+sameRows label df p =+    TestLabel label . TestCase $+        assertEqual label (rowsKept df p) (rowsKept df (simplify p))++anonymousAtomsNotMerged :: Test+anonymousAtomsNotMerged =+    sameRows "distinct udfs over one column are separate atoms" ds $+        F.and+            (F.lift (* 10) d .> F.lit (3 :: Double))+            (F.lift id d .< F.lit (2 :: Double))+  where+    ds = D.fromColumns [("d", D.fromList [1, 2, 3 :: Double])]+    d = F.double "d"++fromMaybeTrueNotFalseOnNull :: Test+fromMaybeTrueNotFalseOnNull =+    sameRows "fromMaybe True keeps null rows" ms $+        F.and+            (F.fromMaybe True (m .< F.lit (5 :: Double)))+            (F.fromMaybe True (m .> F.lit (10 :: Double)))+  where+    ms =+        D.fromColumns+            [("m", D.fromList [Just 1, Nothing, Just 20 :: Maybe Double])]+    m = F.maybeDouble "m"++-- The six-sum moment shape must not see through a user function.+momentsRespectUdf :: Test+momentsRespectUdf =+    TestLabel "moment fast path keeps udf in sum" . TestCase $+        assertEqual "" [60, 150] (D.columnAsList (F.double "sx") out)+  where+    df =+        D.fromColumns+            [ ("k", D.fromList [1, 1, 1, 2, 2, 2 :: Int])+            , ("x", D.fromList [1, 2, 3, 4, 5, 6 :: Double])+            , ("y", D.fromList [6, 5, 4, 3, 2, 1 :: Double])+            ]+    x = F.double "x"+    y = F.double "y"+    out =+        D.sortBy [D.Asc (F.int "k")] $+            D.aggregate+                [ F.count x `as` "n"+                , F.sum (F.lift (* 10) x) `as` "sx"+                , F.sum y `as` "sy"+                , F.sum (x * x) `as` "sxx"+                , F.sum (y * y) `as` "syy"+                , F.sum (x * y) `as` "sxy"+                ]+                (D.groupBy ["k"] df)++-- A function is builtin iff its expression has a provable structural key.+hasKey :: (Columnable a) => Expr a -> Bool+hasKey = (/= Nothing) . structuralKey++-- Every non-higher-order function in DataFrame.Functions is a builtin.+everyFunctionIsBuiltin :: Test+everyFunctionIsBuiltin =+    TestList+        [ TestLabel fn (TestCase (assertBool fn ok))+        | (fn, ok) <- builtins+        ]+  where+    x = F.int "x"+    d = F.double "d"+    m = F.maybeInt "m"+    t = F.text "t"+    b = F.bool "b"+    l = F.col @[Int] "l"+    day = F.col @Day "day"+    builtins =+        [ ("cast", hasKey (F.cast @Int "x"))+        , ("castWithDefault", hasKey (F.castWithDefault (0 :: Int) "x"))+        , ("castEither", hasKey (F.castEither @Int "x"))+        , ("unsafeCast", hasKey (F.unsafeCast @Int "x"))+        , ("castExpr", hasKey (F.castExpr @Int t))+        , ("castExprWithDefault", hasKey (F.castExprWithDefault (0 :: Int) t))+        , ("castExprEither", hasKey (F.castExprEither @Int t))+        , ("unsafeCastExpr", hasKey (F.unsafeCastExpr @Int t))+        , ("toDouble", hasKey (F.toDouble x))+        , ("div", hasKey (F.div x x))+        , ("mod", hasKey (F.mod x x))+        , ("eq", hasKey (F.eq x x))+        , ("lt", hasKey (F.lt x x))+        , ("gt", hasKey (F.gt x x))+        , ("leq", hasKey (F.leq x x))+        , ("geq", hasKey (F.geq x x))+        , ("and", hasKey (F.and b b))+        , ("or", hasKey (F.or b b))+        , ("not", hasKey (F.not b))+        , ("count", hasKey (F.count x))+        , ("countDistinct", hasKey (F.countDistinct x))+        , ("countAll", hasKey F.countAll)+        , ("collect", hasKey (F.collect x))+        , ("mode", hasKey (F.mode x))+        , ("minimum", hasKey (F.minimum x))+        , ("maximum", hasKey (F.maximum x))+        , ("sum", hasKey (F.sum x))+        , ("sumMaybe", hasKey (F.sumMaybe m))+        , ("mean", hasKey (F.mean x))+        , ("topK", hasKey (F.topK 2 x))+        , ("bottomK", hasKey (F.bottomK 2 x))+        , ("meanMaybe", hasKey (F.meanMaybe m))+        , ("variance", hasKey (F.variance x))+        , ("median", hasKey (F.median x))+        , ("medianMaybe", hasKey (F.medianMaybe m))+        , ("percentile", hasKey (F.percentile 50 d))+        , ("stddev", hasKey (F.stddev x))+        , ("stddevMaybe", hasKey (F.stddevMaybe m))+        , ("zScore", hasKey (F.zScore d))+        , ("pow", hasKey (F.pow x 2))+        , ("relu", hasKey (F.relu x))+        , ("min", hasKey (F.min x x))+        , ("max", hasKey (F.max x x))+        , ("toMaybe", hasKey (F.toMaybe x))+        , ("fromMaybe", hasKey (F.fromMaybe 0 m))+        , ("isJust", hasKey (F.isJust m))+        , ("isNothing", hasKey (F.isNothing m))+        , ("isNull", hasKey (F.isNull m))+        , ("isNotNull", hasKey (F.isNotNull m))+        , ("fromJust", hasKey (F.fromJust m))+        , ("recode", hasKey (F.recode [(1 :: Int, 'a')] x))+        , ("recodeWithDefault", hasKey (F.recodeWithDefault 'z' [(1 :: Int, 'a')] x))+        ,+            ( "recodeWithCondition"+            , hasKey (F.recodeWithCondition (F.lit (0 :: Int)) [((.> F.lit @Int 1), 1)] x)+            )+        , ("firstOrNothing", hasKey (F.firstOrNothing l))+        , ("lastOrNothing", hasKey (F.lastOrNothing l))+        , ("contains", hasKey (F.contains "a" t))+        , ("length", hasKey (F.length t))+        , ("splitOn", hasKey (F.splitOn "," t))+        , ("match", hasKey (F.match "a+" t))+        , ("matchAll", hasKey (F.matchAll "a+" t))+        , ("parseDate", hasKey (F.parseDate @Day "%Y" t))+        , ("daysBetween", hasKey (F.daysBetween day day))+        , ("over", hasKey (F.over ["k"] (F.sum x)))+        ]++-- Functions taking arbitrary Haskell functions can only be anonymous.+higherOrderAreAnonymous :: Test+higherOrderAreAnonymous =+    TestList+        [ TestLabel fn (TestCase (assertBool fn (Prelude.not ok)))+        | (fn, ok) <- anonymous+        ]+  where+    x = F.int "x"+    m = F.maybeInt "m"+    anonymous =+        [ ("lift", hasKey (F.lift (+ 1) x))+        , ("lift2", hasKey (F.lift2 (+) x x))+        , ("nullLift", hasKey (F.nullLift (+ 1) m))+        , ("nullLift2", hasKey (F.nullLift2 (+) m x))+        , ("reduce", hasKey (F.reduce x (0 :: Int) (+)))+        , ("whenPresent", hasKey (F.whenPresent (+ 1) m))+        , ("whenBothPresent", hasKey (F.whenBothPresent (+) m m))+        , ("bind", hasKey (F.bind (Just . (+ 1)) m))+        ]++tests :: [Test]+tests =+    [ udfNamedLikeBuiltin+    , udfOtherName+    , anonymousNotEqual+    , namedEqual+    , namedVsBuiltin+    , fromMaybeDefaults+    , powVariants+    , anonymousAtomsNotMerged+    , fromMaybeTrueNotFalseOnNull+    , momentsRespectUdf+    , everyFunctionIsBuiltin+    , higherOrderAreAnonymous+    ]

@@ -94,8 +94,8 @@ aggParityFor n =     TestCase $         let df = caseFrame n-            v = F.col @Int "v"-            vd = F.col @Double "vd"+            v = F.int "v"+            vd = F.double "vd"             aggs =                 [ "vsum" F..= F.sum v                 , "vmean" F..= F.mean vd@@ -132,19 +132,19 @@ momentResult ::     ([T.Text] -> DD.DataFrame -> DD.GroupedDataFrame) -> Int -> DD.DataFrame momentResult grp n =-    let dv1 = F.toDouble (F.col @Int "v1")-        dv2 = F.toDouble (F.col @Int "v2")+    let dv1 = F.toDouble (F.int "v1")+        dv2 = F.toDouble (F.int "v2")         df =             D.derive "v2v2" (dv2 * dv2) $                 D.derive "v1v1" (dv1 * dv1) $                     D.derive "v1v2" (dv1 * dv2) (momentFrame n)         aggs =-            [ "n" F..= F.count (F.col @Int "v1")+            [ "n" F..= F.count (F.int "v1")             , "sx" F..= F.sum dv1             , "sy" F..= F.sum dv2-            , "sxy" F..= F.sum (F.col @Double "v1v2")-            , "sxx" F..= F.sum (F.col @Double "v1v1")-            , "syy" F..= F.sum (F.col @Double "v2v2")+            , "sxy" F..= F.sum (F.double "v1v2")+            , "sxx" F..= F.sum (F.double "v1v1")+            , "syy" F..= F.sum (F.double "v2v2")             ]      in D.aggregate aggs (grp ["id2", "id4"] df) @@ -176,7 +176,7 @@ directGroupAggMatches n =     TestCase $         let df = caseFrame n-            v = F.col @Int "v"+            v = F.int "v"             aggs =                 [ "vsum" F..= F.sum v                 , "vcount" F..= F.count v@@ -222,7 +222,7 @@             | otherwise = T.pack ("k" ++ show ((i * 7919) `mod` 40000))         df0 =             D.fromColumns [("k", packedCol (map key [0 .. n - 1])), mkInt "v" [0 .. n - 1]]-        df = if filtered then D.filterWhere (F.col @T.Text "k" ./=. "") df0 else df0+        df = if filtered then D.filter (F.text "k" ./=. "") df0 else df0     assertBool         "skewed packed text: par == seq"         (sameGrouping (groupByPar ["k"] df) (groupBySeq ["k"] df))

@@ -22,7 +22,7 @@  -- A frame with one derived column "z". withZ :: D.DataFrame-withZ = D.derive "z" (F.col @Int "x" + F.col "y") base+withZ = D.derive "z" (F.int "x" + F.col "y") base  -- Inserting a new column must not wipe provenance of existing derived columns. insertPreservesProvenance :: Test@@ -40,7 +40,7 @@ insertOverwriteDropsOwnExpr :: Test insertOverwriteDropsOwnExpr =     let-        df2 = D.derive "w" (F.col @Int "x") withZ+        df2 = D.derive "w" (F.int "x") withZ         df3 = D.insertColumn "z" (DI.fromList [99 :: Int]) df2      in         TestCase $ do@@ -63,7 +63,7 @@ -- Multiple derives accumulate. deriveManyTracksAll :: Test deriveManyTracksAll =-    let df = D.derive "w" (F.col @Int "x") withZ+    let df = D.derive "w" (F.int "x") withZ      in TestCase             ( assertEqual                 "two derive calls should leave two expressions"@@ -74,7 +74,7 @@ -- Re-deriving a column replaces its expression and keeps the count stable. deriveOverwriteReplacesExpression :: Test deriveOverwriteReplacesExpression =-    let df = D.derive "z" (F.col @Int "y") withZ -- overwrite z+    let df = D.derive "z" (F.int "y") withZ -- overwrite z      in TestCase             ( assertEqual                 "re-deriving z should not duplicate the entry"@@ -85,7 +85,7 @@ -- deriveWithExpr should also track the expression. deriveWithExprTracksExpression :: Test deriveWithExprTracksExpression =-    let (_, df) = D.deriveWithExpr @Int "z" (F.col @Int "x" + F.col "y") base+    let (_, df) = D.deriveWithExpr @Int "z" (F.int "x" + F.col "y") base      in TestCase             ( assertBool                 "deriveWithExpr should record z in derivingExpressions"@@ -123,7 +123,7 @@  semiGroupPreservesBoth :: Test semiGroupPreservesBoth =-    let dfW = D.derive "w" (F.col @Int "y") base+    let dfW = D.derive "w" (F.int "y") base         merged = withZ <> dfW      in TestCase             ( assertEqual@@ -135,8 +135,8 @@ -- Left frame wins when both sides have an expression for the same column. semiGroupLeftBias :: Test semiGroupLeftBias =-    let dfLeft = D.derive "z" (F.col @Int "x") base-        dfRight = D.derive "z" (F.col @Int "y") base+    let dfLeft = D.derive "z" (F.int "x") base+        dfRight = D.derive "z" (F.int "y") base         merged = dfLeft <> dfRight      in TestCase             ( assertEqual@@ -164,7 +164,7 @@  horizontalMergePreservesLeft :: Test horizontalMergePreservesLeft =-    let dfW = D.derive "w" (F.col @Int "y") base+    let dfW = D.derive "w" (F.int "y") base         extra = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])]         merged = dfW ||| extra      in TestCase@@ -179,7 +179,7 @@         dfW =             D.derive                 "w"-                (F.col @Int "y")+                (F.int "y")                 (D.fromColumns [("y", DI.fromList [2 .. 6 :: Int])])         merged = extra ||| dfW      in TestCase@@ -193,7 +193,7 @@     let dfZ = withZ         dfW = D.fromColumns [("q", DI.fromList [0 :: Int, 0, 0, 0, 0])]         -- give dfW a derived column on a separate base-        dfWD = D.derive "w" (F.col @Int "q") dfW+        dfWD = D.derive "w" (F.int "q") dfW         merged = dfZ ||| dfWD      in TestCase             ( assertEqual

@@ -125,8 +125,8 @@ basicTypedRoundTrip :: Test basicTypedRoundTrip = TestCase $ do     let df :: DT.TypedDataFrame OrderSchema-        df = DT.fromRecordsTyped orderSample-    case DT.toRecordsTyped df of+        df = DT.fromRecords orderSample+    case DT.toRecords df of         Left e -> assertFailure (T.unpack e)         Right xs -> assertEqual "typed round-trip" orderSample xs @@ -173,8 +173,8 @@ schemaNameOverride = TestCase $ do     let xs = [Item 1 9.99, Item 2 19.99]         df :: DT.TypedDataFrame ItemsSchema-        df = DT.fromRecordsTyped xs-    case DT.toRecordsTyped df of+        df = DT.fromRecords xs+    case DT.toRecords df of         Left e -> assertFailure (T.unpack e)         Right ys -> assertEqual "schema-name override round-trip" xs ys @@ -323,7 +323,7 @@                 , Order 4 "us" 25.0                 ]         big =-            D.filterWhere+            D.filter                 (orderAmount .>. F.lit @Double 15.0 .&&. orderRegion .==. F.lit @T.Text "us")                 df     case D.toRecords big :: Either T.Text [Order] of@@ -345,14 +345,14 @@     assertEqual         "accessor composes in derive expression"         [20.0, 40.0]-        (D.columnAsList (D.col @Double "double_amount") df')+        (D.columnAsList (D.double "double_amount") df')  labelColumnFilter :: Test labelColumnFilter = TestCase $ do     let df :: DT.TypedDataFrame OrderSchema-        df = DT.fromRecordsTyped orderSample-        usOnly = DT.filterWhere (#region DT..==. "us") df-    case DT.toRecordsTyped usOnly of+        df = DT.fromRecords orderSample+        usOnly = DT.filter (#region DT..==. "us") df+    case DT.toRecords usOnly of         Left e -> assertFailure (T.unpack e)         Right xs ->             assertEqual@@ -424,11 +424,11 @@     df <- D.readParquet "./tests/data/alltypes_plain.parquet"     assertEqual         "prefixed accessor reads its parquet column"-        (D.columnAsList (D.col @Double "double_col") df)+        (D.columnAsList (D.double "double_col") df)         (D.columnAsList alltypesDoubleCol df)     assertEqual         "snake_case parquet column becomes a camelCased accessor"-        (D.columnAsList (D.col @T.Text "date_string_col") df)+        (D.columnAsList (D.text "date_string_col") df)         (D.columnAsList alltypesDateStringCol df)  tests :: [Test]

@@ -80,7 +80,7 @@             (\n -> [if b /= 0 then 2 * (a `div` b) else 0 | (a, b) <- zip (xs n) (ys n)])         , rows             "a null in the branch not chosen is ignored"-            (If (x .>. F.lit 0) (F.col @(Maybe Int) "m") (F.lit (Just 0)))+            (If (x .>. F.lit 0) (F.maybeInt "m") (F.lit (Just 0)))             (\n -> [if a > 0 then m else Just 0 | (a, m) <- zip (xs n) (maybes n)])         , rows             "nested conditions"@@ -103,14 +103,14 @@             )         , rows             "text branches"-            (If (x .>. F.lit 0) (F.lift T.toUpper (F.col @T.Text "name")) (F.lit "none"))+            (If (x .>. F.lit 0) (F.lift T.toUpper (F.text "name")) (F.lit "none"))             (\n -> [if a > 0 then T.toUpper t else "none" | (a, t) <- zip (xs n) (names n)])         , rows             "nullable branches"             ( If                 (x .>. F.lit 0)-                (F.lift (fmap (+ 1)) (F.col @(Maybe Int) "m"))-                (F.col @(Maybe Int) "m")+                (F.lift (fmap (+ 1)) (F.maybeInt "m"))+                (F.maybeInt "m")             )             (\n -> [if a > 0 then fmap (+ 1) m else m | (a, m) <- zip (xs n) (maybes n)])         , rows@@ -132,7 +132,7 @@                     ( either                         (const True)                         (const False)-                        (eval 60 (If (F.lit True) x (F.col @Int "absent")))+                        (eval 60 (If (F.lit True) x (F.int "absent")))                     )                 )             , TestCase@@ -141,7 +141,7 @@                     ( either                         (const True)                         (const False)-                        (eval 60 (If (F.col @Bool "flag") x y))+                        (eval 60 (If (F.bool "flag") x y))                     )                 )             ]

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  module Operations.SetOps where @@ -11,7 +10,7 @@  -- | Sort by the integer key so hash-ordered set results compare stably. sortByA :: D.DataFrame -> D.DataFrame-sortByA = D.sortBy [D.Asc (F.col @Int "A")]+sortByA = D.sortBy [D.Asc (F.int "A")]  dfA :: D.DataFrame dfA =

@@ -49,7 +49,7 @@ shufflePreservesData =     let gen = mkStdGen 1234         shuffled = shuffle gen testDataFrame-        sortedShuffled = D.sortBy [D.Asc (D.col @Int "numbers")] shuffled+        sortedShuffled = D.sortBy [D.Asc (D.int "numbers")] shuffled      in TestCase             (assertEqual "sort recovers initial numbers" testDataFrame sortedShuffled) 

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  module Operations.Sort where @@ -42,7 +41,7 @@                 , ("test2", DI.fromList ['a' .. 'z'])                 ]             )-            (D.sortBy [D.Asc (F.col @Int "test1")] testData)+            (D.sortBy [D.Asc (F.int "test1")] testData)         )  sortByDescendingWAI :: Test@@ -55,7 +54,7 @@                 , ("test2", DI.fromList $ reverse ['a' .. 'z'])                 ]             )-            (D.sortBy [D.Desc (F.col @Int "test1")] testData)+            (D.sortBy [D.Desc (F.int "test1")] testData)         )  sortByTwoColumns :: Test@@ -64,7 +63,7 @@         ( assertEqual             "Sorting moreTestData (which is already sorted) is idempotent."             moreTestData-            (D.sortBy [D.Asc (F.col @Int "test1"), D.Asc (F.col @Int "test2")] moreTestData)+            (D.sortBy [D.Asc (F.int "test1"), D.Asc (F.int "test2")] moreTestData)         )  sortByOneColumnAscOneColumnDesc :: Test@@ -77,7 +76,7 @@                 , ("test2", DI.fromList $ [10 :: Int, 9 .. 1] ++ [10, 9 .. 1])                 ]             )-            (D.sortBy [D.Asc (F.col @Int "test1"), D.Desc (F.col @Int "test2")] moreTestData)+            (D.sortBy [D.Asc (F.int "test1"), D.Desc (F.int "test2")] moreTestData)         )  sortByColumnDoesNotExist :: Test@@ -86,7 +85,7 @@         ( assertExpectException             "[Error Case]"             (D.columnsNotFound ["test0"] "sortBy" (D.columnNames testData))-            (print $ D.sortBy [D.Asc (F.col @Int "test0")] testData)+            (print $ D.sortBy [D.Asc (F.int "test0")] testData)         )  compoundTestData :: D.DataFrame@@ -106,7 +105,7 @@                 , ("b", DI.fromList ([10, 20, 30, 40, 50] :: [Int]))                 ]             )-            (D.sortBy [D.Asc (F.col @Int "a" + F.col @Int "b")] compoundTestData)+            (D.sortBy [D.Asc (F.int "a" + F.int "b")] compoundTestData)         )  sortByCompoundExpressionDescending :: Test@@ -119,7 +118,7 @@                 , ("b", DI.fromList ([50, 40, 30, 20, 10] :: [Int]))                 ]             )-            (D.sortBy [D.Desc (F.col @Int "b" - F.col @Int "a")] compoundTestData)+            (D.sortBy [D.Desc (F.int "b" - F.int "a")] compoundTestData)         )  sortByCompoundMixedWithBareColumn :: Test@@ -133,7 +132,7 @@                 ]             )             ( D.sortBy-                [D.Asc (F.col @Int "a" * 2), D.Desc (F.col @Int "b")]+                [D.Asc (F.int "a" * 2), D.Desc (F.int "b")]                 compoundTestData             )         )@@ -150,7 +149,7 @@             )             ( print $                 D.sortBy-                    [D.Asc (F.col @Int "nope" + F.col @Int "a")]+                    [D.Asc (F.int "nope" + F.int "a")]                     compoundTestData             )         )

@@ -40,7 +40,7 @@     let n = 300000         ks = map (subtract 500) (draws 7 1000 n)         df = D.fromColumns [("k", DI.fromList ks), ("v", DI.fromList [0 .. n - 1 :: Int])]-        ord = if desc then D.Desc (F.col @Int "k") else D.Asc (F.col @Int "k")+        ord = if desc then D.Desc (F.int "k") else D.Asc (F.int "k")         expected = if desc then stableOrder Down ks else stableOrder id ks     assertEqual         ("Int sort desc=" ++ show desc)@@ -54,7 +54,7 @@     assertEqual         "Int32 sort"         (stableOrder id ks)-        (sortedV (df & D.sortBy [D.Asc (F.col @Int32 "k")]))+        (sortedV (df & D.sortBy [D.Asc (F.int32 "k")]))  -- -0.0 and 0.0 compare equal, so they keep row order. doubleSort :: Test@@ -67,11 +67,11 @@     assertEqual         "Double sort"         (stableOrder id ks)-        (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")]))+        (sortedV (df & D.sortBy [D.Asc (F.double "k")]))     assertEqual         "Double sort desc"         (stableOrder Down ks)-        (sortedV (df & D.sortBy [D.Desc (F.col @Double "k")]))+        (sortedV (df & D.sortBy [D.Desc (F.double "k")]))  -- A NaN column keeps the comparison sort; this only checks it still sorts. nanSort :: Test@@ -81,7 +81,7 @@     assertEqual         "NaN sort length"         4-        (length (sortedV (df & D.sortBy [D.Asc (F.col @Double "k")])))+        (length (sortedV (df & D.sortBy [D.Asc (F.double "k")])))  maskCases :: [Test] maskCases =

@@ -99,7 +99,7 @@                 [ ("label", Col.fromList (replicate 50 ("A" :: T.Text) ++ replicate 50 "B"))                 , ("val", Col.fromList ([1 .. 100] :: [Int]))                 ]-        (tr, va) = D.stratifiedSplit (mkStdGen 314) 0.7 (D.col @T.Text "label") df+        (tr, va) = D.stratifiedSplit (mkStdGen 314) 0.7 (D.text "label") df      in fst (dataframeDimensions tr) + fst (dataframeDimensions va) == 100  strataDf :: DataFrame@@ -112,7 +112,7 @@ unit_stratifiedSample_full :: Test unit_stratifiedSample_full =     TestCase $-        let sampled = D.stratifiedSample (mkStdGen 42) 1.0 (D.col @T.Text "label") strataDf+        let sampled = D.stratifiedSample (mkStdGen 42) 1.0 (D.text "label") strataDf          in assertEqual                 "p=1.0 preserves row count"                 (fst $ dataframeDimensions strataDf)@@ -121,7 +121,7 @@ unit_stratifiedSplit_rowCount :: Test unit_stratifiedSplit_rowCount =     TestCase $-        let (tr, va) = D.stratifiedSplit (mkStdGen 99) 0.8 (D.col @T.Text "label") strataDf+        let (tr, va) = D.stratifiedSplit (mkStdGen 99) 0.8 (D.text "label") strataDf          in assertEqual                 "train+validation == total"                 (fst $ dataframeDimensions strataDf)@@ -135,7 +135,7 @@                     [ ("label", Col.fromList (["A", "A", "A", "A", "A", "B"] :: [T.Text]))                     , ("val", Col.fromList ([1 .. 6] :: [Int]))                     ]-            (tr, va) = D.stratifiedSplit (mkStdGen 7) 0.8 (D.col @T.Text "label") tinyDf+            (tr, va) = D.stratifiedSplit (mkStdGen 7) 0.8 (D.text "label") tinyDf          in assertEqual                 "single-row stratum: no rows lost"                 (fst $ dataframeDimensions tinyDf)@@ -164,7 +164,7 @@                         )                     , ("val", Col.fromList ([1 .. aCount + bCount] :: [Int]))                     ]-            (tr, va) = D.stratifiedSplit (mkStdGen 42) 0.8 (D.col @T.Text "label") df+            (tr, va) = D.stratifiedSplit (mkStdGen 42) 0.8 (D.text "label") df             origProp = labelProportion "label" "A" df             trProp = labelProportion "label" "A" tr             vaProp = labelProportion "label" "A" va

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  -- | contains and length on packed text agree with the Text functions per row. module Operations.TextKernels (tests) where@@ -49,7 +48,7 @@             ("contains " ++ show needle)             (map (T.isInfixOf needle) rows)             ( D.columnAsList-                (F.col @Bool "m")+                (F.bool "m")                 (D.derive "m" (F.contains needle (F.col "t")) frame)             ) @@ -59,7 +58,7 @@         assertEqual             "length"             (map T.length rows)-            (D.columnAsList (F.col @Int "l") (D.derive "l" (F.length (F.col "t")) frame))+            (D.columnAsList (F.int "l") (D.derive "l" (F.length (F.col "t")) frame))  tests :: [Test] tests =

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  -- | 'topBy' returns the window a full 'sortBy' would, for every key type. module Operations.TopBy (tests) where@@ -22,14 +21,14 @@  orders :: [(String, [D.SortOrder])] orders =-    [ ("int asc", [D.Asc (F.col @Int "n")])-    , ("int desc", [D.Desc (F.col @Int "n")])-    , ("text asc", [D.Asc (F.col @T.Text "t")])+    [ ("int asc", [D.Asc (F.int "n")])+    , ("int desc", [D.Desc (F.int "n")])+    , ("text asc", [D.Asc (F.text "t")])     ,         ( "double desc, text asc"-        , [D.Desc (F.col @Double "d"), D.Asc (F.col @T.Text "t")]+        , [D.Desc (F.double "d"), D.Asc (F.text "t")]         )-    , ("expr", [D.Asc (F.col @Int "n" * 3 - F.col @Int "id")])+    , ("expr", [D.Asc (F.int "n" * 3 - F.int "id")])     ]  windows :: [(Int, Int)]

@@ -55,11 +55,11 @@         ]  vi, vj :: Expr Int-vi = F.col @Int "vi"-vj = F.col @Int "vj"+vi = F.int "vi"+vj = F.int "vj"  vd :: Expr Double-vd = F.col @Double "vd"+vd = F.double "vd"  aggSets :: [[(T.Text, SomeExpr)]] aggSets =

@@ -33,8 +33,8 @@             "Global median filter removes global outliers"             7             ( purchases-                & D.filterWhere-                    (F.col @Double "amount" .<=. F.median (F.col @Double "amount") * 10)+                & D.filter+                    (F.double "amount" .<=. F.median (F.double "amount") * 10)                 & D.nRows             )         )@@ -50,12 +50,12 @@   where     actual =         purchases-            & D.filterWhere-                ( F.col @Double "amount"-                    .<=. F.over ["country"] (F.median (F.col @Double "amount"))+            & D.filter+                ( F.double "amount"+                    .<=. F.over ["country"] (F.median (F.double "amount"))                     * 10                 )-            & D.sortBy [D.Asc (F.col @T.Text "country"), D.Asc (F.col @Double "amount")]+            & D.sortBy [D.Asc (F.text "country"), D.Asc (F.double "amount")]     expected =         D.fromColumns             [@@ -76,18 +76,18 @@   where     globalResult =         purchases-            & D.filterWhere-                (F.col @Double "amount" .<=. F.median (F.col @Double "amount") * 3)-            & D.sortBy [D.Asc (F.col @T.Text "country"), D.Asc (F.col @Double "amount")]+            & D.filter+                (F.double "amount" .<=. F.median (F.double "amount") * 3)+            & D.sortBy [D.Asc (F.text "country"), D.Asc (F.double "amount")]      overResult =         purchases-            & D.filterWhere-                ( F.col @Double "amount"-                    .<=. F.over ["country"] (F.median (F.col @Double "amount"))+            & D.filter+                ( F.double "amount"+                    .<=. F.over ["country"] (F.median (F.double "amount"))                     * 3                 )-            & D.sortBy [D.Asc (F.col @T.Text "country"), D.Asc (F.col @Double "amount")]+            & D.sortBy [D.Asc (F.text "country"), D.Asc (F.double "amount")]  overMeanDeriveTest :: Test overMeanDeriveTest =@@ -105,8 +105,8 @@             ]     result =         simpleData-            & D.derive "group_mean" (F.over ["group"] (F.mean (F.col @Double "value")))-            & D.sortBy [D.Asc (F.col @T.Text "group"), D.Asc (F.col @Double "value")]+            & D.derive "group_mean" (F.over ["group"] (F.mean (F.double "value")))+            & D.sortBy [D.Asc (F.text "group"), D.Asc (F.double "value")]     -- A mean = 15.0, B mean = 60.0     expectedMeans = [15.0, 15.0, 60.0, 60.0, 60.0] :: [Double]     actualMeans = case DI.getColumn "group_mean" result of@@ -129,8 +129,8 @@             ]     result =         simpleData-            & D.derive "group_sum" (F.over ["group"] (F.sum (F.col @Int "value")))-            & D.sortBy [D.Asc (F.col @T.Text "group"), D.Asc (F.col @Int "value")]+            & D.derive "group_sum" (F.over ["group"] (F.sum (F.int "value")))+            & D.sortBy [D.Asc (F.text "group"), D.Asc (F.int "value")]     expectedSums = [30, 30, 300, 300] :: [Int]     actualSums = case DI.getColumn "group_sum" result of         Nothing -> error "group_sum column not found"@@ -152,8 +152,8 @@             ]     result =         simpleData-            & D.derive "group_count" (F.over ["group"] (F.count (F.col @Int "value")))-            & D.sortBy [D.Asc (F.col @T.Text "group"), D.Asc (F.col @Int "value")]+            & D.derive "group_count" (F.over ["group"] (F.count (F.int "value")))+            & D.sortBy [D.Asc (F.text "group"), D.Asc (F.int "value")]     expectedCounts = [3, 3, 3, 2, 2] :: [Int]     actualCounts = case DI.getColumn "group_count" result of         Nothing -> error "group_count column not found"@@ -177,8 +177,8 @@         simpleData             & D.derive                 "deviation"-                (F.col @Double "value" - F.over ["group"] (F.mean (F.col @Double "value")))-            & D.sortBy [D.Asc (F.col @T.Text "group"), D.Asc (F.col @Double "value")]+                (F.double "value" - F.over ["group"] (F.mean (F.double "value")))+            & D.sortBy [D.Asc (F.text "group"), D.Asc (F.double "value")]     expectedDeviations = [-5.0, 5.0, -50.0, 50.0] :: [Double]     actualDeviations = case DI.getColumn "deviation" result of         Nothing -> error "deviation column not found"@@ -194,14 +194,14 @@             actual         )   where-    amount = F.col @Double "amount"-    discount = F.col @Double "discount"+    amount = F.double "amount"+    discount = F.double "discount"     actual =         purchases-            & D.filterWhere (amount .<=. F.median amount * 10)+            & D.filter (amount .<=. F.median amount * 10)             & D.groupBy ["country"]             & D.aggregate [F.sum (amount - discount) `as` "total"]-            & D.sortBy [D.Asc (F.col @T.Text "country")]+            & D.sortBy [D.Asc (F.text "country")]     -- Global median 60 and threshold 600 remove 5000 and 800.     expected =         D.fromColumns@@ -218,14 +218,14 @@             actual         )   where-    amount = F.col @Double "amount"-    discount = F.col @Double "discount"+    amount = F.double "amount"+    discount = F.double "discount"     actual =         purchases-            & D.filterWhere (amount .<=. F.over ["country"] (F.median amount) * 10)+            & D.filter (amount .<=. F.over ["country"] (F.median amount) * 10)             & D.groupBy ["country"]             & D.aggregate [F.sum (amount - discount) `as` "total"]-            & D.sortBy [D.Asc (F.col @T.Text "country")]+            & D.sortBy [D.Asc (F.text "country")]     expected =         D.fromColumns             [ ("country", DI.fromList (["France", "UK", "US"] :: [T.Text]))

@@ -1,5 +1,4 @@ {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  -- | Packed-text columns must behave identically to boxed Text columns. module PackedTextMigration (tests) where@@ -52,8 +51,8 @@                 [ ("color", mk ["red", "red", "blue", "blue", "green", "green"])                 , ("y", DI.fromList ([0, 0, 1, 1, 2, 2] :: [Double]))                 ]-        packedTree = fit defaultTreeConfig (D.col @Double "y") (df packed)-        boxedTree = fit defaultTreeConfig (D.col @Double "y") (df boxed)+        packedTree = fit defaultTreeConfig (D.double "y") (df packed)+        boxedTree = fit defaultTreeConfig (D.double "y") (df boxed)     assertEqual         "tree on packed-text feature == tree on boxed-text feature"         (D.prettyPrint (predict boxedTree))@@ -67,8 +66,8 @@                 [ ("x", DI.fromList ([1, 2, 3, 4, 5, 6] :: [Double]))                 , ("label", mk ["a", "a", "a", "b", "b", "b"])                 ]-        packedClf = fit defaultTreeConfig (D.col @T.Text "label") (df packed)-        boxedClf = fit defaultTreeConfig (D.col @T.Text "label") (df boxed)+        packedClf = fit defaultTreeConfig (D.text "label") (df packed)+        boxedClf = fit defaultTreeConfig (D.text "label") (df boxed)     assertEqual         "classifier with packed-text target == boxed-text target"         (D.prettyPrint (predict boxedClf))

@@ -0,0 +1,646 @@+{-# LANGUAGE DataKinds #-}+{-# LANGUAGE OverloadedStrings #-}+{-# LANGUAGE TypeApplications #-}++-- | Every operation, run through eager, typed, lazy and typed lazy APIs.+module Parity.FourWay (tests) where++import qualified Data.Text as T+import System.Random (StdGen, mkStdGen)++import qualified DataFrame as D+import qualified DataFrame.Functions as F+import qualified DataFrame.Internal.Column as DI+import qualified DataFrame.Lazy as L+import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF+import qualified DataFrame.Typed.Lazy as TL++import Test.HUnit++type Orders =+    '[ '("cid", Int)+     , '("region", T.Text)+     , '("amount", Double)+     ]++type Customers =+    '[ '("cid", Int)+     , '("region", T.Text)+     , '("tier", T.Text)+     ]++type Users = '[ '("uid", Int), '("name", T.Text)]++ordersDF :: D.DataFrame+ordersDF =+    D.fromColumns+        [ ("cid", DI.fromList [1, 2, 2, 3, 5, 1 :: Int])+        , ("region", DI.fromList ["eu", "us", "us", "eu", "us", "us" :: T.Text])+        , ("amount", DI.fromList [10, 20, 5, 7.5, 3, 1 :: Double])+        ]++customersDF :: D.DataFrame+customersDF =+    D.fromColumns+        [ ("cid", DI.fromList [1, 2, 3, 4 :: Int])+        , ("region", DI.fromList ["eu", "us", "eu", "eu" :: T.Text])+        , ("tier", DI.fromList ["gold", "free", "free", "gold" :: T.Text])+        ]++usersDF :: D.DataFrame+usersDF =+    D.fromColumns+        [ ("uid", DI.fromList [1, 2, 4 :: Int])+        , ("name", DI.fromList ["ann", "bo", "cy" :: T.Text])+        ]++type Nullable = '[ '("k", Int), '("v", Maybe Double), '("w", Maybe Int)]++nullableDF :: D.DataFrame+nullableDF =+    D.fromColumns+        [ ("k", DI.fromList [1, 2, 3, 4, 5 :: Int])+        , ("v", DI.fromList [Just 1.5, Nothing, Just 3, Nothing, Just 5 :: Maybe Double])+        , ("w", DI.fromList [Nothing, Nothing, Just 3, Nothing, Just 1 :: Maybe Int])+        ]++nullableT :: DT.TypedDataFrame Nullable+nullableT = either (error . show) id (DT.freezeWithError nullableDF)++tln :: TL.TypedLazyDataFrame Nullable+tln = TL.fromDataFrame nullableT++ordersT :: DT.TypedDataFrame Orders+ordersT = either (error . show) id (DT.freezeWithError ordersDF)++customersT :: DT.TypedDataFrame Customers+customersT = either (error . show) id (DT.freezeWithError customersDF)++usersT :: DT.TypedDataFrame Users+usersT = either (error . show) id (DT.freezeWithError usersDF)++-- | Compare all four results against the eager one.+fourWay ::+    String ->+    D.DataFrame ->+    DT.TypedDataFrame cols ->+    L.LazyDataFrame ->+    TL.TypedLazyDataFrame cols ->+    Test+fourWay lbl eager typed lazy typedLazy = TestCase $ do+    sameFrame (lbl ++ ": typed == eager") eager (DT.thaw typed)+    lazyDf <- L.run lazy+    sameFrame (lbl ++ ": lazy == eager") eager lazyDf+    typedLazyDf <- TL.run typedLazy+    sameFrame (lbl ++ ": typed lazy == eager") eager (DT.thaw typedLazyDf)++-- | Equal values and equal column order ('==' ignores the order).+sameFrame :: String -> D.DataFrame -> D.DataFrame -> Assertion+sameFrame lbl expected actual =+    assertEqual+        lbl+        (D.columnNames expected, expected)+        (D.columnNames actual, actual)++-- | Two-row batches, so streaming ops cross batch boundaries.+lazyOf :: D.DataFrame -> L.LazyDataFrame+lazyOf df = (L.fromDataFrame df){L.batchSize = 2}++lo, lc, lu, ln :: L.LazyDataFrame+lo = lazyOf ordersDF+lc = lazyOf customersDF+lu = lazyOf usersDF+ln = lazyOf nullableDF++tlo :: TL.TypedLazyDataFrame Orders+tlo = TL.fromDataFrame ordersT++tlc :: TL.TypedLazyDataFrame Customers+tlc = TL.fromDataFrame customersT++tlu :: TL.TypedLazyDataFrame Users+tlu = TL.fromDataFrame usersT++joins :: Test+joins =+    TestList+        [ fourWay+            "innerJoin multi-key"+            (D.innerJoin ["cid", "region"] ordersDF customersDF)+            (DT.innerJoin @'["cid", "region"] ordersT customersT)+            (L.innerJoin ["cid", "region"] lo lc)+            (TL.innerJoin @'["cid", "region"] tlo tlc)+        , fourWay+            "join INNER"+            (D.join D.INNER ["cid"] ordersDF customersDF)+            (DT.join @'DT.INNER @'["cid"] ordersT customersT)+            (L.join L.INNER ["cid"] lo lc)+            (TL.join @'TL.INNER @'["cid"] tlo tlc)+        , fourWay+            "leftJoin"+            (D.leftJoin ["cid"] ordersDF customersDF)+            (DT.leftJoin @'["cid"] ordersT customersT)+            (L.leftJoin ["cid"] lo lc)+            (TL.leftJoin @'["cid"] tlo tlc)+        , fourWay+            "rightJoin"+            (D.rightJoin ["cid"] ordersDF customersDF)+            (DT.rightJoin @'["cid"] ordersT customersT)+            (L.rightJoin ["cid"] lo lc)+            (TL.rightJoin @'["cid"] tlo tlc)+        , fourWay+            "fullOuterJoin"+            (D.fullOuterJoin ["cid"] ordersDF customersDF)+            (DT.fullOuterJoin @'["cid"] ordersT customersT)+            (L.fullOuterJoin ["cid"] lo lc)+            (TL.fullOuterJoin @'["cid"] tlo tlc)+        , fourWay+            "joinOn differing key names"+            (D.joinOn D.LEFT [("cid", "uid")] ordersDF usersDF)+            (DT.joinOn @'DT.LEFT @'[ '("cid", "uid")] ordersT usersT)+            (L.joinOn L.LEFT [("cid", "uid")] lo lu)+            (TL.joinOn @'TL.LEFT @'[ '("cid", "uid")] tlo tlu)+        ]++groupAndSort :: Test+groupAndSort =+    TestList+        [ fourWay+            "groupBy + aggregate + sortBy"+            ( ordersDF+                D.|> D.groupBy ["region"]+                D.|> D.aggregate+                    [ F.sum (F.double "amount") `D.as` "total"+                    , F.count (F.double "amount") `D.as` "n"+                    ]+                D.|> D.sortBy [D.Asc (F.text "region")]+            )+            ( ordersT+                DT.|> DT.groupBy @'["region"]+                DT.|> DT.aggregate+                    ( DT.as @"n" (DTF.count (DT.col @"amount"))+                        . DT.as @"total" (DTF.sum (DT.col @"amount"))+                    )+                DT.|> DT.sortBy [DT.Asc (DT.col @"region")]+            )+            ( lo+                L.|> L.groupBy ["region"]+                L.|> L.aggregate+                    [ F.sum (F.double "amount") `L.as` "total"+                    , F.count (F.double "amount") `L.as` "n"+                    ]+                L.|> L.sortBy [L.Asc (L.text "region")]+            )+            ( tlo+                TL.|> TL.groupBy @'["region"]+                TL.|> TL.aggregate+                    ( TL.as @"n" (DTF.count (TL.col @"amount"))+                        . TL.as @"total" (DTF.sum (TL.col @"amount"))+                    )+                TL.|> TL.sortBy [TL.Asc (TL.col @"region")]+            )+        , fourWay+            "sortBy expression, filter, derive, select, take"+            ( ordersDF+                D.|> D.filter (F.double "amount" D..>. 2)+                D.|> D.derive "twice" (F.double "amount" * 2)+                D.|> D.sortBy [D.Desc (F.double "amount" * (-1))]+                D.|> D.select ["cid", "twice"]+                D.|> D.take 3+            )+            ( ordersT+                DT.|> DT.filter (DT.col @"amount" DT..>. 2)+                DT.|> DT.derive @"twice" (DT.col @"amount" * 2)+                DT.|> DT.sortBy [DT.Desc (DT.col @"amount" * (-1))]+                DT.|> DT.select @'["cid", "twice"]+                DT.|> DT.take 3+            )+            ( lo+                L.|> L.filter (F.double "amount" L..>. 2)+                L.|> L.derive "twice" (F.double "amount" * 2)+                L.|> L.sortBy [L.Desc (F.double "amount" * (-1))]+                L.|> L.select ["cid", "twice"]+                L.|> L.take 3+            )+            ( tlo+                TL.|> TL.filter (TL.col @"amount" TL..>. 2)+                TL.|> TL.derive @"twice" (TL.col @"amount" * 2)+                TL.|> TL.sortBy [TL.Desc (TL.col @"amount" * (-1))]+                TL.|> TL.select @'["cid", "twice"]+                TL.|> TL.take 3+            )+        ]++-- | Untyped eager vs untyped lazy.+untypedPair :: String -> D.DataFrame -> L.LazyDataFrame -> Test+untypedPair lbl eager lazy = TestCase $ do+    lazyDf <- L.run lazy+    sameFrame (lbl ++ ": lazy == eager") eager lazyDf++-- | Typed eager vs typed lazy.+typedPair ::+    String -> DT.TypedDataFrame cols -> TL.TypedLazyDataFrame cols -> Test+typedPair lbl typed typedLazy = TestCase $ do+    typedLazyDf <- TL.run typedLazy+    sameFrame (lbl ++ ": typed lazy == typed") (DT.thaw typed) (DT.thaw typedLazyDf)++-- | A value computed four ways.+valueFourWay :: (Eq a, Show a) => String -> a -> a -> IO a -> IO a -> Test+valueFourWay lbl eager typed lazy typedLazy = TestCase $ do+    assertEqual (lbl ++ ": typed") eager typed+    lazyV <- lazy+    assertEqual (lbl ++ ": lazy") eager lazyV+    typedLazyV <- typedLazy+    assertEqual (lbl ++ ": typed lazy") eager typedLazyV++g :: StdGen+g = mkStdGen 42++rowOps :: Test+rowOps =+    TestList+        [ fourWay+            "filterBy"+            (D.filterBy (> 4) (F.double "amount") ordersDF)+            (DT.filterBy (> 4) (DT.col @"amount") ordersT)+            (L.filterBy (> 4) (F.double "amount") lo)+            (TL.filterBy (> 4) (TL.col @"amount") tlo)+        , fourWay+            "filterNothing"+            (D.filterNothing "v" nullableDF)+            (DT.filterNothing @"v" nullableT)+            (L.filterNothing "v" ln)+            (TL.filterNothing @"v" tln)+        , fourWay+            "filterAllNothing"+            (D.filterAllNothing (D.exclude ["k"] nullableDF))+            (DT.filterAllNothing (DT.exclude @'["k"] nullableT))+            (L.filterAllNothing (L.exclude ["k"] ln))+            (TL.filterAllNothing (TL.exclude @'["k"] tln))+        , fourWay+            "filterJust"+            (D.filterJust "v" nullableDF)+            (DT.filterJust @"v" nullableT)+            (L.filterJust "v" ln)+            (TL.filterJust @"v" tln)+        , fourWay+            "filterAllJust"+            (D.filterAllJust nullableDF)+            (DT.filterAllJust nullableT)+            (L.filterAllJust ln)+            (TL.filterAllJust tln)+        , fourWay+            "drop"+            (D.drop 3 ordersDF)+            (DT.drop 3 ordersT)+            (L.drop 3 lo)+            (TL.drop 3 tlo)+        , fourWay+            "takeLast"+            (D.takeLast 2 ordersDF)+            (DT.takeLast 2 ordersT)+            (L.takeLast 2 lo)+            (TL.takeLast 2 tlo)+        , fourWay+            "dropLast"+            (D.dropLast 2 ordersDF)+            (DT.dropLast 2 ordersT)+            (L.dropLast 2 lo)+            (TL.dropLast 2 tlo)+        , fourWay+            "range"+            (D.range (1, 4) ordersDF)+            (DT.range (1, 4) ordersT)+            (L.range (1, 4) lo)+            (TL.range (1, 4) tlo)+        , fourWay+            "range clamped"+            (D.range (-2, 100) ordersDF)+            (DT.range (-2, 100) ordersT)+            (L.range (-2, 100) lo)+            (TL.range (-2, 100) tlo)+        , fourWay+            "cube"+            (D.cube (3, 2) ordersDF)+            (DT.cube (3, 2) ordersT)+            (L.cube (3, 2) lo)+            (TL.cube (3, 2) tlo)+        , fourWay+            "topBy"+            (D.topBy 1 3 [D.Desc (F.double "amount")] ordersDF)+            (DT.topBy 1 3 [DT.Desc (DT.col @"amount")] ordersT)+            (L.topBy 1 3 [L.Desc (F.double "amount")] lo)+            (TL.topBy 1 3 [TL.Desc (TL.col @"amount")] tlo)+        , fourWay+            "distinct"+            (D.distinct (D.select ["cid", "region"] ordersDF))+            (DT.distinct (DT.select @'["cid", "region"] ordersT))+            (L.distinct (L.select ["cid", "region"] lo))+            (TL.distinct (TL.select @'["cid", "region"] tlo))+        , fourWay+            "sample"+            (D.sample g 0.5 ordersDF)+            (DT.sample g 0.5 ordersT)+            (L.sample g 0.5 lo)+            (TL.sample g 0.5 tlo)+        , fourWay+            "shuffle"+            (D.shuffle g ordersDF)+            (DT.shuffle g ordersT)+            (L.shuffle g lo)+            (TL.shuffle g tlo)+        , fourWay+            "selectRows"+            (D.selectRows [4, 0, 2] ordersDF)+            (DT.selectRows [4, 0, 2] ordersT)+            (L.selectRows [4, 0, 2] lo)+            (TL.selectRows [4, 0, 2] tlo)+        , fourWay+            "stratifiedSample"+            (D.stratifiedSample g 0.5 (F.text "region") ordersDF)+            (DT.stratifiedSample g 0.5 (DT.col @"region") ordersT)+            (L.stratifiedSample g 0.5 (F.text "region") lo)+            (TL.stratifiedSample g 0.5 (TL.col @"region") tlo)+        , fourWay+            "randomSplit fst"+            (fst (D.randomSplit g 0.5 ordersDF))+            (fst (DT.randomSplit g 0.5 ordersT))+            (fst (L.randomSplit g 0.5 lo))+            (fst (TL.randomSplit g 0.5 tlo))+        , fourWay+            "randomSplit snd"+            (snd (D.randomSplit g 0.5 ordersDF))+            (snd (DT.randomSplit g 0.5 ordersT))+            (snd (L.randomSplit g 0.5 lo))+            (snd (TL.randomSplit g 0.5 tlo))+        , fourWay+            "stratifiedSplit snd"+            (snd (D.stratifiedSplit g 0.5 (F.text "region") ordersDF))+            (snd (DT.stratifiedSplit g 0.5 (DT.col @"region") ordersT))+            (snd (L.stratifiedSplit g 0.5 (F.text "region") lo))+            (snd (TL.stratifiedSplit g 0.5 (TL.col @"region") tlo))+        , fourWay+            "kFolds last"+            (last (D.kFolds g 3 ordersDF))+            (last (DT.kFolds g 3 ordersT))+            (last (L.kFolds g 3 lo))+            (last (TL.kFolds g 3 tlo))+        ]++columnOps :: Test+columnOps =+    TestList+        [ fourWay+            "deriveMany"+            ( D.deriveMany+                [ "a2" D..= (F.double "amount" * 2)+                , "c1" D..= (F.int "cid" + 1)+                ]+                ordersDF+            )+            ( DT.deriveMany+                (DT.as @"c1" (DT.col @"cid" + 1) . DT.as @"a2" (DT.col @"amount" * 2))+                ordersT+            )+            ( L.deriveMany+                [ "a2" L..= (F.double "amount" * 2)+                , "c1" L..= (F.int "cid" + 1)+                ]+                lo+            )+            ( TL.deriveMany+                (TL.as @"c1" (TL.col @"cid" + 1) . TL.as @"a2" (TL.col @"amount" * 2))+                tlo+            )+        , fourWay+            "deriveWithExpr"+            (snd (D.deriveWithExpr "a2" (F.double "amount" * 2) ordersDF))+            (snd (DT.deriveWithExpr @"a2" (DT.col @"amount" * 2) ordersT))+            (snd (L.deriveWithExpr "a2" (F.double "amount" * 2) lo))+            (snd (TL.deriveWithExpr @"a2" (TL.col @"amount" * 2) tlo))+        , fourWay+            "exclude"+            (D.exclude ["region"] ordersDF)+            (DT.exclude @'["region"] ordersT)+            (L.exclude ["region"] lo)+            (TL.exclude @'["region"] tlo)+        , fourWay+            "rename then filter on new name"+            (D.filter (F.double "amt" D..>. 4) (D.rename "amount" "amt" ordersDF))+            (DT.filter (DT.col @"amt" DT..>. 4) (DT.rename @"amount" @"amt" ordersT))+            (L.filter (F.double "amt" L..>. 4) (L.rename "amount" "amt" lo))+            (TL.filter (TL.col @"amt" TL..>. 4) (TL.rename @"amount" @"amt" tlo))+        , fourWay+            "renameMany then select"+            (D.select ["c"] (D.renameMany [("cid", "c"), ("region", "r")] ordersDF))+            (DT.select @'["c"] (DT.renameMany @'[ '("cid", "c"), '("region", "r")] ordersT))+            (L.select ["c"] (L.renameMany [("cid", "c"), ("region", "r")] lo))+            (TL.select @'["c"] (TL.renameMany @'[ '("cid", "c"), '("region", "r")] tlo))+        , fourWay+            "cloneColumn"+            (D.cloneColumn "cid" "cid2" ordersDF)+            (DT.cloneColumn @"cid" @"cid2" ordersT)+            (L.cloneColumn "cid" "cid2" lo)+            (TL.cloneColumn @"cid" @"cid2" tlo)+        , fourWay+            "dropColumn / exclude"+            (D.exclude ["amount"] ordersDF)+            (DT.dropColumn @"amount" ordersT)+            (L.exclude ["amount"] lo)+            (TL.dropColumn @"amount" tlo)+        , fourWay+            "insert"+            (D.insert "n" [10 .. 15 :: Int] ordersDF)+            (DT.insert @"n" [10 .. 15 :: Int] ordersT)+            (L.insert "n" [10 .. 15 :: Int] lo)+            (TL.insert @"n" [10 .. 15 :: Int] tlo)+        , fourWay+            "insertWithDefault"+            (D.insertWithDefault 0 "n" [1, 2 :: Int] ordersDF)+            (DT.insertWithDefault @"n" 0 [1, 2 :: Int] ordersT)+            (L.insertWithDefault 0 "n" [1, 2 :: Int] lo)+            (TL.insertWithDefault @"n" 0 [1, 2 :: Int] tlo)+        , fourWay+            "apply"+            (D.apply (show :: Int -> String) "cid" ordersDF)+            (DT.apply @"cid" (show :: Int -> String) ordersT)+            (L.apply (show :: Int -> String) "cid" lo)+            (TL.apply @"cid" (show :: Int -> String) tlo)+        , fourWay+            "applyMany"+            (D.applyMany (* (2 :: Double)) ["amount"] ordersDF)+            (DT.applyMany @'["amount"] (* 2) ordersT)+            (L.applyMany (* (2 :: Double)) ["amount"] lo)+            (TL.applyMany @'["amount"] (* 2) tlo)+        , fourWay+            "applyWhere"+            ( D.applyWhere+                (== ("us" :: T.Text))+                "region"+                (+ (100 :: Double))+                "amount"+                ordersDF+            )+            (DT.applyWhere @"region" @"amount" (== "us") (+ 100) ordersT)+            (L.applyWhere (== ("us" :: T.Text)) "region" (+ (100 :: Double)) "amount" lo)+            (TL.applyWhere @"region" @"amount" (== "us") (+ 100) tlo)+        , fourWay+            "applyAtIndex"+            (D.applyAtIndex 3 (negate :: Double -> Double) "amount" ordersDF)+            (DT.applyAtIndex @"amount" 3 negate ordersT)+            (L.applyAtIndex 3 (negate :: Double -> Double) "amount" lo)+            (TL.applyAtIndex @"amount" 3 negate tlo)+        , fourWay+            "imputeWith"+            (D.imputeWith F.mean (F.maybeDouble "v") nullableDF)+            (DT.imputeWith DTF.mean (DT.col @"v") nullableT)+            (L.imputeWith F.mean (F.maybeDouble "v") ln)+            (TL.imputeWith DTF.mean (TL.col @"v") tln)+        , untypedPair+            "impute (untyped)"+            (D.impute (F.maybeDouble "v") 0 nullableDF)+            (L.impute (F.maybeDouble "v") 0 ln)+        , typedPair+            "impute (typed)"+            (DT.impute @"v" 0 nullableT)+            (TL.impute @"v" 0 tln)+        , typedPair+            "replaceColumn"+            (DT.replaceColumn @"amount" (DT.col @"amount" * 3) ordersT)+            (TL.replaceColumn @"amount" (TL.col @"amount" * 3) tlo)+        ]++mergeOps :: Test+mergeOps =+    TestList+        [ fourWay+            "append"+            (ordersDF <> ordersDF)+            (DT.append ordersT ordersT)+            (lo <> lo)+            (TL.append tlo tlo)+        , fourWay+            "(|||)"+            (D.select ["cid"] ordersDF D.||| D.select ["amount"] ordersDF)+            (DT.select @'["cid"] ordersT DT.||| DT.select @'["amount"] ordersT)+            (L.select ["cid"] lo L.||| L.select ["amount"] lo)+            (TL.select @'["cid"] tlo TL.||| TL.select @'["amount"] tlo)+        , fourWay+            "union"+            (D.union ordersDF (D.take 2 ordersDF))+            (DT.union ordersT (DT.take 2 ordersT))+            (L.union lo (L.take 2 lo))+            (TL.union tlo (TL.take 2 tlo))+        , fourWay+            "intersect"+            (D.intersect ordersDF (D.take 2 ordersDF))+            (DT.intersect ordersT (DT.take 2 ordersT))+            (L.intersect lo (L.take 2 lo))+            (TL.intersect tlo (TL.take 2 tlo))+        , fourWay+            "difference"+            (D.difference ordersDF (D.take 2 ordersDF))+            (DT.difference ordersT (DT.take 2 ordersT))+            (L.difference lo (L.take 2 lo))+            (TL.difference tlo (TL.take 2 tlo))+        , fourWay+            "symmetricDifference"+            (D.symmetricDifference ordersDF (D.drop 3 ordersDF))+            (DT.symmetricDifference ordersT (DT.drop 3 ordersT))+            (L.symmetricDifference lo (L.drop 3 lo))+            (TL.symmetricDifference tlo (TL.drop 3 tlo))+        , fourWay+            "fold"+            (D.fold (\n -> D.filter (F.int "cid" D../=. F.lit n)) [1, 2] ordersDF)+            (DT.fold (\n -> DT.filter (DT.col @"cid" DT../=. DT.lit n)) [1, 2] ordersT)+            (L.fold (\n -> L.filter (F.int "cid" L../=. F.lit n)) [1, 2] lo)+            (TL.fold (\n -> TL.filter (TL.col @"cid" TL../=. TL.lit n)) [1, 2] tlo)+        ]++summaries :: Test+summaries =+    TestList+        [ untypedPair "summarize" (D.summarize ordersDF) (L.summarize lo)+        , untypedPair "summarize (typed)" (DT.summarize ordersT) (TL.summarize tlo)+        , untypedPair+            "describeColumns"+            (D.describeColumns ordersDF)+            (L.describeColumns lo)+        , untypedPair+            "frequencies"+            (D.frequencies (F.text "region") ordersDF)+            (L.frequencies (F.text "region") lo)+        , untypedPair+            "frequencies (typed)"+            (DT.frequencies (DT.col @"region") ordersT)+            (TL.frequencies (TL.col @"region") tlo)+        ]++values :: Test+values =+    TestList+        [ valueFourWay+            "mean"+            (D.mean (F.double "amount") ordersDF)+            (DT.mean (DT.col @"amount") ordersT)+            (L.mean (F.double "amount") lo)+            (TL.mean (TL.col @"amount") tlo)+        , valueFourWay+            "sum"+            (D.sum (F.int "cid") ordersDF)+            (DT.sum (DT.col @"cid") ordersT)+            (L.sum (F.int "cid") lo)+            (TL.sum (TL.col @"cid") tlo)+        , valueFourWay+            "median of a filtered query"+            (D.median (F.double "amount") (D.filter (F.int "cid" D..>. 1) ordersDF))+            (DT.median (DT.col @"amount") (DT.filter (DT.col @"cid" DT..>. 1) ordersT))+            (L.median (F.double "amount") (L.filter (F.int "cid" L..>. 1) lo))+            (TL.median (TL.col @"amount") (TL.filter (TL.col @"cid" TL..>. 1) tlo))+        , valueFourWay+            "dimensions"+            (D.dimensions ordersDF)+            (DT.dimensions ordersT)+            (L.dimensions lo)+            (TL.dimensions tlo)+        , valueFourWay+            "columnNames"+            (D.columnNames ordersDF)+            (DT.columnNames ordersT)+            (L.columnNames lo)+            (TL.columnNames tlo)+        , valueFourWay+            "valueCounts"+            (D.valueCounts (F.text "region") ordersDF)+            (DT.valueCounts (DT.col @"region") ordersT)+            (L.valueCounts (F.text "region") lo)+            (TL.valueCounts (TL.col @"region") tlo)+        , valueFourWay+            "columnAsList"+            (D.columnAsList (F.double "amount") ordersDF)+            (DT.columnAsList @"amount" ordersT)+            (L.columnAsList (F.double "amount") lo)+            (TL.columnAsList @"amount" tlo)+        , valueFourWay+            "correlation"+            (D.correlation "cid" "amount" ordersDF)+            (DT.correlation @"cid" @"amount" ordersT)+            (L.correlation "cid" "amount" lo)+            (TL.correlation @"cid" @"amount" tlo)+        , valueFourWay+            "toCsv"+            (D.toCsv ordersDF)+            (DT.toCsv ordersT)+            (L.toCsv lo)+            (TL.toCsv tlo)+        , valueFourWay+            "null"+            (D.null ordersDF)+            (DT.null ordersT)+            (L.null lo)+            (TL.null tlo)+        ]++tests :: [Test]+tests = [joins, groupAndSort, rowOps, columnOps, mergeOps, summaries, values]

@@ -99,7 +99,7 @@     TestCase         ( assertEqual             "allTypesPlainSnappy"-            (D.filter (F.col @Int32 "id") (`elem` [6, 7]) allTypes)+            (D.filterBy (`elem` [6, 7]) (F.int32 "id") allTypes)             (unsafePerformIO (readParquet "./tests/data/alltypes_plain.snappy.parquet"))         ) @@ -108,7 +108,7 @@     TestCase         ( assertEqual             "allTypesPlainSnappy"-            (D.filter (F.col @Int32 "id") (`elem` [0, 1]) allTypes)+            (D.filterBy (`elem` [0, 1]) (F.int32 "id") allTypes)             (unsafePerformIO (readParquet "./tests/data/alltypes_dictionary.parquet"))         ) @@ -154,7 +154,7 @@                         , D.predicate =                             Just                                 ( F.geq-                                    (F.col @Int32 "id")+                                    (F.int32 "id")                                     (F.lit (6 :: Int32))                                 )                         }@@ -177,7 +177,7 @@                         , D.predicate =                             Just                                 ( F.geq-                                    (F.col @Int32 "id")+                                    (F.int32 "id")                                     (F.lit (6 :: Int32))                                 )                         }@@ -253,7 +253,7 @@                             , D.predicate =                                 Just                                     ( F.geq-                                        (F.col @Int32 "id")+                                        (F.int32 "id")                                         (F.lit (6 :: Int32))                                     )                             }

@@ -109,10 +109,36 @@         "weird column name present as a data key"         (Data.Maybe.isJust (lookupKey weird row0))     assertEqual-        "encoding references the weird field name"-        (Just (String weird))+        "encoding references the weird field name, backslash escaped"+        (Just (String "we\"ir\\\\d"))         (jpath ["encoding", "x", "field"] spec) +dottedFieldName :: Test+dottedFieldName = TestCase $ do+    let name = "sepal.length"+        df =+            D.fromColumns+                [ (name, D.fromList ([5.1, 4.9] :: [Double]))+                , ("variety", D.fromList (["a", "b"] :: [T.Text]))+                ]+        spec =+            C.toVegaSpec+                ( C.chart df+                    & C.mark C.Bar+                    & C.enc C.X (col @T.Text "variety")+                    & C.enc C.Y (col @Double name)+                    & C.aggregateOn C.Y C.Sum+                )+        row0 = fromMaybe Null (dataValues spec V.!? 0)+    assertEqual+        "data key keeps the dot"+        (Just (toJSON (5.1 :: Double)))+        (lookupKey name row0)+    assertEqual+        "field reference escapes the dot so Vega-Lite does not nest"+        (Just (String "sepal\\.length"))+        (jpath ["encoding", "y", "field"] spec)+ computedExpr :: Test computedExpr = TestCase $ do     let spec =@@ -204,13 +230,34 @@                 )     assertEqual "typed spec equals untyped spec" specU specT +nullableBecomesNull :: Test+nullableBecomesNull = TestCase $ do+    let df =+            D.fromColumns+                [ ("g", D.fromList ([Just "a", Nothing, Just "b"] :: [Maybe T.Text]))+                , ("n", D.fromList ([Just 1, Nothing, Just 3] :: [Maybe Double]))+                ]+        spec =+            C.toVegaSpec+                ( C.chart df+                    & C.mark C.Bar+                    & C.enc C.X (col @(Maybe T.Text) "g")+                    & C.enc C.Y (col @(Maybe Double) "n")+                )+        row1 = fromMaybe Null (dataValues spec V.!? 1)+    assertEqual "null category is JSON null" (Just Null) (lookupKey "g" row1)+    assertEqual "null number is JSON null, not 0" (Just Null) (lookupKey "n" row1)+    assertEqual "row count kept" 3 (V.length (dataValues spec))+ tests :: [Test] tests =     [ TestLabel "Plotting.fieldTypeInference" fieldTypeInference     , TestLabel "Plotting.boxIsBoxplot" boxIsBoxplot     , TestLabel "Plotting.legacyBoxIsBoxplot" legacyBoxIsBoxplot     , TestLabel "Plotting.nanBecomesNull" nanBecomesNull+    , TestLabel "Plotting.nullableBecomesNull" nullableBecomesNull     , TestLabel "Plotting.escapingSafe" escapingSafe+    , TestLabel "Plotting.dottedFieldName" dottedFieldName     , TestLabel "Plotting.computedExpr" computedExpr     , TestLabel "Plotting.includeZeroChart" includeZeroChart     , TestLabel "Plotting.includeZeroMergesWithLog" includeZeroMergesWithLog

@@ -1,6 +1,5 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-}  -- | Specification for simplify: each case compares the full expression. module Simplify (tests) where@@ -28,31 +27,31 @@     [ simplifiesTo         "and lower bounds keeps max"         ( F.and-            (F.col @Double "age" .> F.lit (20 :: Double))-            (F.col @Double "age" .> F.lit (25 :: Double))+            (F.double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (25 :: Double))         )-        (F.col @Double "age" .> F.lit (25 :: Double))+        (F.double "age" .> F.lit (25 :: Double))     , simplifiesTo         "and upper bounds keeps min"         ( F.and-            (F.col @Double "age" .< F.lit (50 :: Double))-            (F.col @Double "age" .< F.lit (40 :: Double))+            (F.double "age" .< F.lit (50 :: Double))+            (F.double "age" .< F.lit (40 :: Double))         )-        (F.col @Double "age" .< F.lit (40 :: Double))+        (F.double "age" .< F.lit (40 :: Double))     , simplifiesTo         "or lower bounds keeps min"         ( F.or-            (F.col @Double "age" .> F.lit (20 :: Double))-            (F.col @Double "age" .> F.lit (25 :: Double))+            (F.double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (25 :: Double))         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     , simplifiesTo         "or upper bounds keeps max"         ( F.or-            (F.col @Double "age" .< F.lit (50 :: Double))-            (F.col @Double "age" .< F.lit (40 :: Double))+            (F.double "age" .< F.lit (50 :: Double))+            (F.double "age" .< F.lit (40 :: Double))         )-        (F.col @Double "age" .< F.lit (50 :: Double))+        (F.double "age" .< F.lit (50 :: Double))     ]  mixedDirection :: [Test]@@ -60,52 +59,52 @@     [ simplifiesTo         "closed interval at a point becomes equality"         ( F.and-            (F.col @Double "age" .>= F.lit (30 :: Double))-            (F.col @Double "age" .<= F.lit (30 :: Double))+            (F.double "age" .>= F.lit (30 :: Double))+            (F.double "age" .<= F.lit (30 :: Double))         )-        (F.col @Double "age" .== F.lit (30 :: Double))+        (F.double "age" .== F.lit (30 :: Double))     , simplifiesTo         "open contradiction becomes False"         ( F.and-            (F.col @Double "age" .> F.lit (30 :: Double))-            (F.col @Double "age" .< F.lit (30 :: Double))+            (F.double "age" .> F.lit (30 :: Double))+            (F.double "age" .< F.lit (30 :: Double))         )         (F.lit False)     , simplifiesTo         "disjoint bounds become False"         ( F.and-            (F.col @Double "age" .> F.lit (30 :: Double))-            (F.col @Double "age" .< F.lit (20 :: Double))+            (F.double "age" .> F.lit (30 :: Double))+            (F.double "age" .< F.lit (20 :: Double))         )         (F.lit False)     , simplifiesTo         "distinct points conjoined become False"         ( F.and-            (F.col @Double "age" .== F.lit (30 :: Double))-            (F.col @Double "age" .== F.lit (40 :: Double))+            (F.double "age" .== F.lit (30 :: Double))+            (F.double "age" .== F.lit (40 :: Double))         )         (F.lit False)     , simplifiesTo         "point inside half-space becomes the point"         ( F.and-            (F.col @Double "age" .== F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (25 :: Double))+            (F.double "age" .== F.lit (30 :: Double))+            (F.double "age" .> F.lit (25 :: Double))         )-        (F.col @Double "age" .== F.lit (30 :: Double))+        (F.double "age" .== F.lit (30 :: Double))     , simplifiesTo         "point outside half-space becomes False"         ( F.and-            (F.col @Double "age" .== F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (40 :: Double))+            (F.double "age" .== F.lit (30 :: Double))+            (F.double "age" .> F.lit (40 :: Double))         )         (F.lit False)     , simplifiesTo         "negation redundant under bound drops"         ( F.and-            (F.col @Double "age" ./= F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (40 :: Double))+            (F.double "age" ./= F.lit (30 :: Double))+            (F.double "age" .> F.lit (40 :: Double))         )-        (F.col @Double "age" .> F.lit (40 :: Double))+        (F.double "age" .> F.lit (40 :: Double))     ]  tautologies :: [Test]@@ -113,22 +112,22 @@     [ simplifiesTo         "integral exhaustive cover becomes True"         ( F.or-            (F.toDouble (F.col @Int "ai") .<= F.lit (30 :: Double))-            (F.toDouble (F.col @Int "ai") .> F.lit (30 :: Double))+            (F.toDouble (F.int "ai") .<= F.lit (30 :: Double))+            (F.toDouble (F.int "ai") .> F.lit (30 :: Double))         )         (F.lit True)     , simplifiesTo         "distinct inequalities cover everything"         ( F.or-            (F.col @Double "age" ./= F.lit (30 :: Double))-            (F.col @Double "age" ./= F.lit (40 :: Double))+            (F.double "age" ./= F.lit (30 :: Double))+            (F.double "age" ./= F.lit (40 :: Double))         )         (F.lit True)     , simplifiesTo         "inequality or equality at same point"         ( F.or-            (F.col @Double "age" ./= F.lit (30 :: Double))-            (F.col @Double "age" .== F.lit (30 :: Double))+            (F.double "age" ./= F.lit (30 :: Double))+            (F.double "age" .== F.lit (30 :: Double))         )         (F.lit True)     ]@@ -138,32 +137,32 @@     [ simplifiesTo         "idempotent and"         ( F.and-            (F.col @Double "age" .> F.lit (20 :: Double))-            (F.col @Double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (20 :: Double))         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     , simplifiesTo         "absorption and over or"         ( F.and-            (F.col @Double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (20 :: Double))             ( F.or-                (F.col @Double "age" .> F.lit (20 :: Double))-                (F.col @Double "hours" .> F.lit (40 :: Double))+                (F.double "age" .> F.lit (20 :: Double))+                (F.double "hours" .> F.lit (40 :: Double))             )         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     , simplifiesTo         "true and unit"-        (F.and (F.lit True) (F.col @Double "hours" .> F.lit (40 :: Double)))-        (F.col @Double "hours" .> F.lit (40 :: Double))+        (F.and (F.lit True) (F.double "hours" .> F.lit (40 :: Double)))+        (F.double "hours" .> F.lit (40 :: Double))     , simplifiesTo         "false and annihilates"-        (F.and (F.lit False) (F.col @Double "hours" .> F.lit (40 :: Double)))+        (F.and (F.lit False) (F.double "hours" .> F.lit (40 :: Double)))         (F.lit False)     , simplifiesTo         "double negation"-        (F.not (F.not (F.col @Double "age" .> F.lit (20 :: Double))))-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.not (F.not (F.double "age" .> F.lit (20 :: Double))))+        (F.double "age" .> F.lit (20 :: Double))     ]  ifCollapse :: [Test]@@ -171,19 +170,19 @@     [ simplifiesTo         "boolean if becomes its condition"         ( F.ifThenElse-            (F.col @Double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (20 :: Double))             (F.lit True)             (F.lit False)         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     , simplifiesTo         "if with equal branches collapses"         ( F.ifThenElse-            (F.col @Double "hours" .> F.lit (40 :: Double))-            (F.col @Double "age" .> F.lit (20 :: Double))-            (F.col @Double "age" .> F.lit (20 :: Double))+            (F.double "hours" .> F.lit (40 :: Double))+            (F.double "age" .> F.lit (20 :: Double))+            (F.double "age" .> F.lit (20 :: Double))         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     ]  multiPass :: [Test]@@ -193,34 +192,34 @@         ( F.and             ( F.and                 ( F.and-                    (F.col @Double "age" .> F.lit (10 :: Double))-                    (F.col @Double "age" .> F.lit (20 :: Double))+                    (F.double "age" .> F.lit (10 :: Double))+                    (F.double "age" .> F.lit (20 :: Double))                 )-                (F.col @Double "age" .> F.lit (30 :: Double))+                (F.double "age" .> F.lit (30 :: Double))             )-            (F.col @Double "age" .> F.lit (40 :: Double))+            (F.double "age" .> F.lit (40 :: Double))         )-        (F.col @Double "age" .> F.lit (40 :: Double))+        (F.double "age" .> F.lit (40 :: Double))     , simplifiesTo         "consolidate then contradiction"         ( F.and             ( F.and-                (F.col @Double "age" .>= F.lit (30 :: Double))-                (F.col @Double "age" .>= F.lit (40 :: Double))+                (F.double "age" .>= F.lit (30 :: Double))+                (F.double "age" .>= F.lit (40 :: Double))             )-            (F.col @Double "age" .<= F.lit (35 :: Double))+            (F.double "age" .<= F.lit (35 :: Double))         )         (F.lit False)     , simplifiesTo         "cascade of contradictions"         ( F.or             ( F.and-                (F.col @Double "age" .> F.lit (30 :: Double))-                (F.col @Double "age" .< F.lit (20 :: Double))+                (F.double "age" .> F.lit (30 :: Double))+                (F.double "age" .< F.lit (20 :: Double))             )             ( F.and-                (F.col @Double "hours" .> F.lit (200 :: Double))-                (F.col @Double "hours" .< F.lit (10 :: Double))+                (F.double "hours" .> F.lit (200 :: Double))+                (F.double "hours" .< F.lit (10 :: Double))             )         )         (F.lit False)@@ -228,21 +227,21 @@         "consolidate enabling idempotence"         ( F.and             ( F.or-                (F.col @Double "age" .> F.lit (20 :: Double))-                (F.col @Double "age" .> F.lit (25 :: Double))+                (F.double "age" .> F.lit (20 :: Double))+                (F.double "age" .> F.lit (25 :: Double))             )             ( F.or-                (F.col @Double "age" .> F.lit (20 :: Double))-                (F.col @Double "age" .> F.lit (30 :: Double))+                (F.double "age" .> F.lit (20 :: Double))+                (F.double "age" .> F.lit (30 :: Double))             )         )-        (F.col @Double "age" .> F.lit (20 :: Double))+        (F.double "age" .> F.lit (20 :: Double))     , simplifiesTo         "de morgan over contradiction"         ( F.not             ( F.and-                (F.col @Double "age" .> F.lit (30 :: Double))-                (F.col @Double "age" .< F.lit (20 :: Double))+                (F.double "age" .> F.lit (30 :: Double))+                (F.double "age" .< F.lit (20 :: Double))             )         )         (F.lit True)@@ -250,12 +249,12 @@         "interior contradiction collapses the conjunction"         ( F.and             ( F.and-                (F.col @Double "age" .> F.lit (10 :: Double))-                (F.col @Double "hours" .> F.lit (40 :: Double))+                (F.double "age" .> F.lit (10 :: Double))+                (F.double "hours" .> F.lit (40 :: Double))             )             ( F.and-                (F.col @Double "age" .> F.lit (30 :: Double))-                (F.col @Double "age" .< F.lit (25 :: Double))+                (F.double "age" .> F.lit (30 :: Double))+                (F.double "age" .< F.lit (25 :: Double))             )         )         (F.lit False)@@ -265,45 +264,45 @@ nullAware =     [ simplifiesTo         "just-literal lower bounds keep max"-        ( (F.col @Int "age" .> F.lit (Just (30 :: Int)))-            .&& (F.col @Int "age" .> F.lit (Just (35 :: Int)))+        ( (F.int "age" .> F.lit (Just (30 :: Int)))+            .&& (F.int "age" .> F.lit (Just (35 :: Int)))         )-        (F.col @Int "age" .> F.lit (Just (35 :: Int)))+        (F.int "age" .> F.lit (Just (35 :: Int)))     , simplifiesTo         "just-literal contradiction over non-null column becomes Just False"-        ( (F.col @Int "age" .> F.lit (Just (30 :: Int)))-            .&& (F.col @Int "age" .< F.lit (Just (20 :: Int)))+        ( (F.int "age" .> F.lit (Just (30 :: Int)))+            .&& (F.int "age" .< F.lit (Just (20 :: Int)))         )         (F.lit (Just False))     , unchanged         "nullable column contradiction stays unknown"-        ( (F.col @(Maybe Int) "w" .> F.lit (Just (30 :: Int)))-            .&& (F.col @(Maybe Int) "w" .< F.lit (Just (20 :: Int)))+        ( (F.maybeInt "w" .> F.lit (Just (30 :: Int)))+            .&& (F.maybeInt "w" .< F.lit (Just (20 :: Int)))         )     , unchanged         "nullable column tautology stays unknown"-        ( (F.col @(Maybe Int) "w" .<= F.lit (Just (30 :: Int)))-            .|| (F.col @(Maybe Int) "w" .> F.lit (Just (30 :: Int)))+        ( (F.maybeInt "w" .<= F.lit (Just (30 :: Int)))+            .|| (F.maybeInt "w" .> F.lit (Just (30 :: Int)))         )     , simplifiesTo         "fromMaybe consolidation keeps tighter"         ( F.and-            (F.fromMaybe False (F.col @(Maybe Double) "w" .<= F.lit (5 :: Double)))-            (F.fromMaybe False (F.col @(Maybe Double) "w" .<= F.lit (3 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .<= F.lit (5 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .<= F.lit (3 :: Double)))         )-        (F.fromMaybe False (F.col @(Maybe Double) "w" .<= F.lit (3 :: Double)))+        (F.fromMaybe False (F.maybeDouble "w" .<= F.lit (3 :: Double)))     , simplifiesTo         "fromMaybe contradiction becomes False"         ( F.and-            (F.fromMaybe False (F.col @(Maybe Double) "w" .> F.lit (30 :: Double)))-            (F.fromMaybe False (F.col @(Maybe Double) "w" .< F.lit (20 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .> F.lit (30 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .< F.lit (20 :: Double)))         )         (F.lit False)     , unchanged         "fromMaybe tautology stays unsimplified"         ( F.or-            (F.fromMaybe False (F.col @(Maybe Double) "w" .<= F.lit (30 :: Double)))-            (F.fromMaybe False (F.col @(Maybe Double) "w" .> F.lit (30 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .<= F.lit (30 :: Double)))+            (F.fromMaybe False (F.maybeDouble "w" .> F.lit (30 :: Double)))         )     ] @@ -312,38 +311,38 @@     [ unchanged         "proper interval is not collapsed"         ( F.and-            (F.col @Double "age" .>= F.lit (20 :: Double))-            (F.col @Double "age" .<= F.lit (65 :: Double))+            (F.double "age" .>= F.lit (20 :: Double))+            (F.double "age" .<= F.lit (65 :: Double))         )     , unchanged         "or with a gap is not a tautology"         ( F.or-            (F.col @Double "age" .<= F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (40 :: Double))+            (F.double "age" .<= F.lit (30 :: Double))+            (F.double "age" .> F.lit (40 :: Double))         )     , unchanged         "two inequalities are not an interval"         ( F.and-            (F.col @Double "age" ./= F.lit (30 :: Double))-            (F.col @Double "age" ./= F.lit (40 :: Double))+            (F.double "age" ./= F.lit (30 :: Double))+            (F.double "age" ./= F.lit (40 :: Double))         )     , unchanged         "cross-column conjunction is left alone"         ( F.and-            (F.col @Double "age" .> F.lit (50 :: Double))-            (F.col @Double "hours" .> F.lit (40 :: Double))+            (F.double "age" .> F.lit (50 :: Double))+            (F.double "hours" .> F.lit (40 :: Double))         )     , unchanged         "double exhaustive cover bails (NaN)"         ( F.or-            (F.col @Double "age" .<= F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (30 :: Double))+            (F.double "age" .<= F.lit (30 :: Double))+            (F.double "age" .> F.lit (30 :: Double))         )     , unchanged         "punctured interval is not a single atom"         ( F.and-            (F.col @Double "age" ./= F.lit (30 :: Double))-            (F.col @Double "age" .> F.lit (20 :: Double))+            (F.double "age" ./= F.lit (30 :: Double))+            (F.double "age" .> F.lit (20 :: Double))         )     ] 

@@ -10,9 +10,12 @@ import System.Random (mkStdGen)  import qualified DataFrame as D+import DataFrame.Expression.Operators ((./=.)) import qualified DataFrame.Functions as F import qualified DataFrame.Internal.Column as DI+import DataFrame.Internal.Expression (eqExpr) import qualified DataFrame.Typed as DT+import qualified DataFrame.Typed.Functions as DTF import qualified DataFrame.Typed.Statistics as TS  import Test.HUnit@@ -38,29 +41,29 @@ statParity :: Test statParity =     TestList-        [ eq "mean" (TS.mean (DT.col @"y") tdf) (D.mean (F.col @Double "y") baseDF)-        , eq "median" (TS.median (DT.col @"y") tdf) (D.median (F.col @Double "y") baseDF)+        [ eq "mean" (TS.mean (DT.col @"y") tdf) (D.mean (F.double "y") baseDF)+        , eq "median" (TS.median (DT.col @"y") tdf) (D.median (F.double "y") baseDF)         , eq             "variance"             (TS.variance (DT.col @"y") tdf)-            (D.variance (F.col @Double "y") baseDF)+            (D.variance (F.double "y") baseDF)         , eq             "stddev"             (TS.standardDeviation (DT.col @"y") tdf)-            (D.standardDeviation (F.col @Double "y") baseDF)-        , eq "sum" (TS.sum (DT.col @"x") tdf) (D.sum (F.col @Int "x") baseDF)+            (D.standardDeviation (F.double "y") baseDF)+        , eq "sum" (TS.sum (DT.col @"x") tdf) (D.sum (F.int "x") baseDF)         , eq             "percentile"             (TS.percentile 50 (DT.col @"y") tdf)-            (D.percentile 50 (F.col @Double "y") baseDF)+            (D.percentile 50 (F.double "y") baseDF)         , eq             "iqr"             (TS.interQuartileRange (DT.col @"y") tdf)-            (D.interQuartileRange (F.col @Double "y") baseDF)+            (D.interQuartileRange (F.double "y") baseDF)         , eq             "skewness"             (TS.skewness (DT.col @"y") tdf)-            (D.skewness (F.col @Double "y") baseDF)+            (D.skewness (F.double "y") baseDF)         , eq             "correlation"             (TS.correlation @"x" @"y" tdf)@@ -74,23 +77,44 @@ exprParity :: Test exprParity =     TestList-        [ deriveEq "pow" (DT.pow (DT.col @"x") 2) (F.pow (F.col @Int "x") 2)-        , deriveEq "relu" (DT.relu (DT.col @"x")) (F.relu (F.col @Int "x"))-        , deriveEq "toMaybe" (DT.toMaybe (DT.col @"x")) (F.toMaybe (F.col @Int "x"))+        [ deriveEq "pow" (DTF.pow (DT.col @"x") 2) (F.pow (F.int "x") 2)+        , deriveEq "relu" (DTF.relu (DT.col @"x")) (F.relu (F.int "x"))+        , deriveEq "toMaybe" (DTF.toMaybe (DT.col @"x")) (F.toMaybe (F.int "x"))         , deriveEq             "min"-            (DT.min (DT.col @"x") (DT.lit 3))-            (F.min (F.col @Int "x") (F.lit @Int 3))+            (DTF.min (DT.col @"x") (DT.lit 3))+            (F.min (F.int "x") (F.lit @Int 3))         , deriveEq             "splitOn"-            (DT.splitOn "a" (DT.col @"g"))-            (F.splitOn "a" (F.col @T.Text "g"))+            (DTF.splitOn "a" (DT.col @"g"))+            (F.splitOn "a" (F.text "g"))         , deriveEq             "recodeWithDefault"-            (DT.recodeWithDefault 0 [(1 :: Int, 100 :: Int)] (DT.col @"x"))-            (F.recodeWithDefault 0 [(1 :: Int, 100 :: Int)] (F.col @Int "x"))+            (DTF.recodeWithDefault 0 [(1 :: Int, 100 :: Int)] (DT.col @"x"))+            (F.recodeWithDefault 0 [(1 :: Int, 100 :: Int)] (F.int "x"))+        , deriveEq+            "eq"+            (DTF.eq (DT.col @"x") (DT.lit 3))+            (F.eq (F.int "x") (F.lit @Int 3))+        , deriveEq+            "and"+            (DTF.and (DTF.lt (DT.col @"x") (DT.lit 5)) (DTF.geq (DT.col @"x") (DT.lit 2)))+            (F.and (F.lt x (F.lit @Int 5)) (F.geq x (F.lit @Int 2)))+        , deriveEq "contains" (DTF.contains "a" (DT.col @"g")) (F.contains "a" g)+        , deriveEq "length" (DTF.length (DT.col @"g")) (F.length g)+        , deriveEq+            "isNull"+            (DTF.isNull (DTF.toMaybe (DT.col @"x")))+            (F.isNull (F.toMaybe x))+        , deriveEq "cast" (DTF.cast @"x" @Int) (F.cast @Int "x")+        , deriveEq+            "castWithDefault"+            (DTF.castWithDefault @"g" (0 :: Int))+            (F.castWithDefault (0 :: Int) "g")         ]   where+    x = F.int "x"+    g = F.text "g"     deriveEq ::         (DI.Columnable a) =>         String -> DT.TExpr S a -> D.Expr a -> Test@@ -102,15 +126,29 @@                 (DT.thaw (DT.derive @"r" te tdf))             ) +-- Typed aggregations build the same expression as their untyped forms.+aggParity :: Test+aggParity =+    TestList+        [ same "countDistinct" (DTF.countDistinct (DT.col @"x")) (F.countDistinct x)+        , same "topK" (DTF.topK 2 (DT.col @"x")) (F.topK 2 x)+        , same "bottomK" (DTF.bottomK 2 (DT.col @"x")) (F.bottomK 2 x)+        , same "not" (DTF.not (DTF.eq (DT.col @"x") 1)) (F.not (F.eq x 1))+        ]+  where+    x = F.int "x"+    same :: (DI.Columnable a) => String -> DT.TExpr S a -> D.Expr a -> Test+    same lbl te ue = TestCase (assertBool lbl (eqExpr (DT.unTExpr te) ue))+ -- apply / valueCounts / horizontal merge. applyParity :: Test applyParity =     TestList         [ TestCase             ( assertEqual-                "applyColumn type-change"+                "apply type-change"                 (D.apply (show :: Int -> String) "x" baseDF)-                (DT.thaw (DT.applyColumn @"x" (show :: Int -> String) tdf))+                (DT.thaw (DT.apply @"x" (show :: Int -> String) tdf))             )         , TestCase             ( assertEqual@@ -121,7 +159,7 @@         , TestCase             ( assertEqual                 "valueCounts"-                (D.valueCounts (F.col @T.Text "g") baseDF)+                (D.valueCounts (F.text "g") baseDF)                 (DT.valueCounts (DT.col @"g") tdf)             )         , TestCase@@ -157,7 +195,7 @@         [ TestCase             ( assertEqual                 "columnAsDoubleVector"-                (extract (D.columnAsDoubleVector (F.col @Double "y") baseDF))+                (extract (D.columnAsDoubleVector (F.double "y") baseDF))                 (DT.columnAsDoubleVector @"y" tdf)             )         , TestCase@@ -195,10 +233,75 @@ numericT :: DT.TypedDataFrame '[ '("x", Int), '("y", Double)] numericT = either (error . show) id (DT.freezeWithError numericDF) +-- Ops added for import-swap parity: same result through either API.+newOpsParity :: Test+newOpsParity =+    TestList+        [ same+            "topBy"+            (D.topBy 1 2 [D.Desc (F.double "y")] baseDF)+            (DT.topBy 1 2 [DT.Desc (DT.col @"y")] tdf)+        , same+            "deriveMany"+            ( D.deriveMany+                [ "x2" D..= (F.int "x" * 2)+                , "y2" D..= (F.double "y" + 1)+                ]+                baseDF+            )+            ( DT.deriveMany+                (DT.as @"y2" (DT.col @"y" + 1) . DT.as @"x2" (DT.col @"x" * 2))+                tdf+            )+        , same+            "join INNER"+            (D.join D.INNER ["x"] leftDF numericDF)+            (DT.join @'DT.INNER @'["x"] leftT numericT)+        , same+            "join LEFT"+            (D.join D.LEFT ["x"] numericDF leftDF)+            (DT.join @'DT.LEFT @'["x"] numericT leftT)+        , same+            "applyInt"+            (D.applyInt (* 10) "x" baseDF)+            (DT.applyInt @"x" (* 10) tdf)+        , same+            "applyDouble"+            (D.applyDouble (> 20) "y" baseDF)+            (DT.applyDouble @"y" (> 20) tdf)+        , same+            "fold"+            (D.fold (\n -> D.filter (F.int "x" ./=. F.lit n)) [2, 4] baseDF)+            (DT.fold (\n -> DT.filter (DT.col @"x" DT../=. DT.lit n)) [2, 4] tdf)+        , TestCase (assertBool "empty is null" (DT.null DT.empty))+        , TestCase (assertEqual "null" (D.null baseDF) (DT.null tdf))+        , TestCase (assertEqual "toCsv" (D.toCsv baseDF) (DT.toCsv tdf))+        , TestCase+            (assertEqual "toMarkdown" (D.toMarkdown baseDF) (DT.toMarkdown tdf))+        , TestCase+            ( assertEqual+                "frame stats via DataFrame.Typed"+                (D.mean (F.double "y") baseDF)+                (DT.mean (DT.col @"y") tdf)+            )+        , TestCase+            ( assertEqual+                "readJSONEither"+                (D.toCsv <$> D.readJSONEither jsonBytes)+                (DT.toCsv <$> DT.readJSONEither @'[ '("a", Maybe Double)] jsonBytes)+            )+        ]+  where+    same :: String -> D.DataFrame -> DT.TypedDataFrame cols -> Test+    same lbl untyped typed = TestCase (assertEqual lbl untyped (DT.thaw typed))+    jsonBytes = "[{\"a\": 1}, {\"a\": 2}]"+ tests :: [Test] tests =-    [ statParity+    [ newOpsParity+    , statParity     , exprParity+    , aggParity     , applyParity     , samplingParity     , accessParity