dataframe 2.1.0.0 → 2.1.0.1

raw patch · 3 files changed

+317/−201 lines, 3 filesdep +deepseqdep ~dataframePVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependencies added: deepseq

Dependency ranges changed: dataframe

API changes (from Hackage documentation)

Files

• README.md +304/−198
• benchmark/Main.hs +11/−0
• dataframe.cabal +2/−3

@@ -1,3 +1,15 @@+<!--+  This file is the runnable scripths source for the project README.+  Every ```haskell block below executes in order in a single shared session+  (later blocks see bindings from earlier blocks), and scripths inserts each+  block's output beneath it as a blockquote.++  Regenerate the rendered README with:+      scripths docs/base_scripts/base_readme.md -o README.md+  (run from the repo root, so the ./data/... paths resolve).++  Blocks tagged ```text are intentional compile-error demos and are NOT run.+--> <h1 align="center">   <a href="https://dataframe.readthedocs.io/en/latest/">     <img width="100" height="100" src="https://raw.githubusercontent.com/mchav/dataframe/master/docs/_static/haskell-logo.svg" alt="dataframe logo">@@ -29,6 +41,8 @@ - **Typed** (`import qualified DataFrame.Typed as T`) — phantom-type schema tracking with compile-time column validation. - **Monadic API** — write your transformation as a self contained pipeline. +> **This README is a runnable [scripths](https://github.com/DataHaskell/scripths) notebook.** Every Haskell block runs top-to-bottom in one shared session against the datasets in [`./data`](./data). Reproduce every output below with `scripths docs/base_scripts/base_readme.md -o README.md` from the repo root.+ ## Why this library?  * Concise, declarative, composable data pipelines using the `|>` pipe operator.@@ -38,148 +52,166 @@  ## Install + ```bash cabal update cabal install dataframe ``` + To use as a dependency in a project: + ``` build-depends: base >= 4, dataframe ``` + Works with GHC 9.4 through 9.12. A custom REPL with all imports pre-loaded is available after installing: + ```bash dataframe ``` + ## Quick Start -Save this as `Example.hs` and run with `cabal run Example.hs`:+Group sales by product and compute totals. The first block carries the+`scripths` cabal directives and the imports shared by the rest of the document;+you can also drop the same code into an `Example.hs` and run it with+`cabal run Example.hs` after adding a `#!/usr/bin/env cabal` header. -```haskell-#!/usr/bin/env cabal-{- cabal:-  build-depends: base >= 4, dataframe--}-{-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE TypeApplications #-} +```haskell+-- cabal: build-depends: dataframe, text+-- cabal: default-extensions: OverloadedStrings, TypeApplications, TemplateHaskell, DataKinds, TypeFamilies, FlexibleInstances, FlexibleContexts, ScopedTypeVariables, DeriveGeneric, UndecidableInstances import qualified DataFrame as D import qualified DataFrame.Functions as F+import qualified DataFrame.Typed as DT import DataFrame.Operators+import Data.Text (Text)+import Data.Int (Int64) -main :: IO ()-main = do-    let sales = D.fromNamedColumns-            [ ("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int])-            , ("amount",  D.fromList [100, 120, 50, 20, 40, 30 :: Int])-            ]+sales = D.fromNamedColumns+    [ ("product", D.fromList [1, 1, 2, 2, 3, 3 :: Int])+    , ("amount",  D.fromList [100, 120, 50, 20, 40, 30 :: Int])+    ] -    -- Group by product and compute totals-    print $ sales-        |> D.groupBy ["product"]-        |> D.aggregate [ F.sum (F.col @Int "amount") `as` "total"-                       , F.count (F.col @Int "amount") `as` "orders"-                       ]+-- Group by product and compute totals+sales+    |> D.groupBy ["product"]+    |> D.aggregate [ F.sum (F.col @Int "amount") `as` "total"+                   , F.count (F.col @Int "amount") `as` "orders"+                   ]+    |> D.toMarkdown' ``` -```-------------------------product | total | orders---------|-------|--------  Int   |  Int  |  Int---------|-------|--------1       | 220   | 2-2       | 70    | 2-3       | 70    | 2-```+> <!-- sabela:mime text/plain -->+> | product<br>Int | total<br>Int | orders<br>Int |+> | ---------------|--------------|-------------- |+> | 1              | 220          | 2             |+> | 3              | 70           | 2             |+> | 2              | 70           | 2             | + Reading from files works the same way: + ```haskell-df <- D.readCsv "data.csv"-df <- D.readParquet "data.parquet"+fileDf <- D.readCsv "./data/housing.csv"+fileDf <- D.readParquet "./data/mtcars.parquet" --- Hugging Face datasets-df <- D.readParquet "hf://datasets/scikit-learn/iris/default/train/0000.parquet"+-- Hugging Face datasets (needs network access):+-- fileDf <- D.readParquet "hf://datasets/scikit-learn/iris/default/train/0000.parquet"++D.dimensions fileDf ``` +> <!-- sabela:mime text/plain -->+> (32,12)++ ## Interactive REPL -The `dataframe` REPL comes with all imports pre-loaded. Here's a typical exploration session:+The `dataframe` REPL comes with all imports pre-loaded. Here's a typical exploration session (each block runs as a cell): + ```haskell-dataframe> df <- D.readCsv "./data/housing.csv"-dataframe> D.dimensions df-(20640, 10)+df <- D.readCsv "./data/housing.csv"+D.dimensions df+``` -dataframe> D.describeColumns df--------------------------------------------------------------------------    Column Name     | ## Non-null Values | ## Null Values |     Type---------------------|--------------------|----------------|--------------        Text        |         Int        |      Int       |     Text---------------------|--------------------|----------------|-------------- total_bedrooms     | 20433              | 207            | Maybe Double- ocean_proximity    | 20640              | 0              | Text- median_house_value | 20640              | 0              | Double- median_income      | 20640              | 0              | Double- households         | 20640              | 0              | Double- population         | 20640              | 0              | Double- total_rooms        | 20640              | 0              | Double- housing_median_age | 20640              | 0              | Double- latitude           | 20640              | 0              | Double- longitude          | 20640              | 0              | Double+> <!-- sabela:mime text/plain -->+> (20640,10)++++```haskell+D.describeColumns df |> D.toMarkdown' ``` -The `:declareColumns` macro generates typed column references from a dataframe, so you can use column names directly in expressions instead of writing `F.col @Double "median_income"` every time:+> <!-- sabela:mime text/plain -->+> | Column Name<br>Text | # Non-null Values<br>Int | # Null Values<br>Int | Type<br>Text |+> | --------------------|--------------------------|----------------------|------------- |+> | total_bedrooms      | 20433                    | 207                  | Maybe Double |+> | ocean_proximity     | 20640                    | 0                    | Text         |+> | median_house_value  | 20640                    | 0                    | Double       |+> | median_income       | 20640                    | 0                    | Double       |+> | households          | 20640                    | 0                    | Double       |+> | population          | 20640                    | 0                    | Double       |+> | total_rooms         | 20640                    | 0                    | Double       |+> | housing_median_age  | 20640                    | 0                    | Double       |+> | latitude            | 20640                    | 0                    | Double       |+> | longitude           | 20640                    | 0                    | Double       | ++The `:declareColumns` macro (`$(D.declareColumns df)` outside the REPL) generates typed column references from a dataframe, so you can use column names directly in expressions instead of writing `F.col @Double "median_income"` every time:++ ```haskell-dataframe> :declareColumns df-"longitude :: Expr Double"-"latitude :: Expr Double"-"housing_median_age :: Expr Double"-"total_rooms :: Expr Double"-"total_bedrooms :: Expr (Maybe Double)"-"population :: Expr Double"-"households :: Expr Double"-"median_income :: Expr Double"-"median_house_value :: Expr Double"-"ocean_proximity :: Expr Text"+$(D.declareColumns df)+``` -dataframe> df |> D.groupBy ["ocean_proximity"]-              |> D.aggregate [F.mean median_house_value `as` "avg_value"]--------------------------------------- ocean_proximity |     avg_value------------------|--------------------      Text       |       Double------------------|-------------------- <1H OCEAN       | 240084.28546409807- INLAND          | 124805.39200122119- ISLAND          | 380440.0- NEAR BAY        | 259212.31179039303- NEAR OCEAN      | 249433.97742663656+> <!-- sabela:mime text/plain -->++++```haskell+df |> D.groupBy ["ocean_proximity"]+   |> D.aggregate [F.mean median_house_value `as` "avg_value"]+   |> D.toMarkdown' ``` +> <!-- sabela:mime text/plain -->+> | ocean_proximity<br>Text | avg_value<br>Double |+> | ------------------------|-------------------- |+> | NEAR BAY                | 259212.31179039303  |+> | NEAR OCEAN              | 249433.97742663656  |+> | INLAND                  | 124805.39200122119  |+> | <1H OCEAN               | 240084.28546409807  |+> | ISLAND                  | 380440.0            |++ Create new columns from existing ones: + ```haskell-dataframe> df |> D.derive "rooms_per_household" (total_rooms / households) |> D.take 3------------------------------------------------------------------------------------------------------------------- longitude | latitude | housing_median_age | total_rooms | ... | ocean_proximity | rooms_per_household------------|----------|--------------------|-------------|-----|-----------------|---------------------  Double   |  Double  |       Double       |   Double    | ... |      Text       |       Double------------|----------|--------------------|-------------|-----|-----------------|--------------------- -122.23   | 37.88    | 41.0               | 880.0       | ... | NEAR BAY        | 6.984126984126984- -122.22   | 37.86    | 21.0               | 7099.0      | ... | NEAR BAY        | 6.238137082601054- -122.24   | 37.85    | 52.0               | 1467.0      | ... | NEAR BAY        | 8.288135593220339+df |> D.derive "rooms_per_household" (total_rooms / households) |> D.take 3 |> D.toMarkdown' ``` +> <!-- sabela:mime text/plain -->+> | longitude<br>Double | latitude<br>Double | housing_median_age<br>Double | total_rooms<br>Double | total_bedrooms<br>Maybe Double | population<br>Double | households<br>Double | median_income<br>Double | median_house_value<br>Double | ocean_proximity<br>Text | rooms_per_household<br>Double |+> | --------------------|--------------------|------------------------------|-----------------------|--------------------------------|----------------------|----------------------|-------------------------|------------------------------|-------------------------|------------------------------ |+> | -122.23             | 37.88              | 41.0                         | 880.0                 | Just 129.0                     | 322.0                | 126.0                | 8.3252                  | 452600.0                     | NEAR BAY                | 6.984126984126984             |+> | -122.22             | 37.86              | 21.0                         | 7099.0                | Just 1106.0                    | 2401.0               | 1138.0               | 8.3014                  | 358500.0                     | NEAR BAY                | 6.238137082601054             |+> | -122.24             | 37.85              | 52.0                         | 1467.0                | Just 190.0                     | 496.0                | 177.0                | 7.2574                  | 352100.0                     | NEAR BAY                | 8.288135593220339             |++ Type mismatches are caught as compile errors — adding a `Double` column to a `Text` column won't silently produce garbage: -```haskell++```text dataframe> df |> D.derive "nonsense" (latitude + ocean_proximity)  <interactive>:14:47: error: [GHC-83865]@@ -191,6 +223,7 @@         '(latitude + ocean_proximity)' ``` + ## Template Haskell  For scripts and projects, Template Haskell can generate column bindings at compile time.@@ -200,14 +233,8 @@ `declareColumnsFromCsvFile` (in `DataFrame.TH`, also re-exported from `DataFrame`) reads your CSV at compile time and generates typed `Expr` bindings for every column: -```haskell-{-# LANGUAGE TemplateHaskell #-}-{-# LANGUAGE OverloadedStrings #-} -import qualified DataFrame as D-import qualified DataFrame.Functions as F-import DataFrame.Operators-+```haskell -- Reads housing.csv at compile time and generates: --   latitude :: Expr Double --   total_rooms :: Expr Double@@ -215,44 +242,63 @@ --   ... one binding per column $(D.declareColumnsFromCsvFile "./data/housing.csv") -main :: IO ()-main = do-    df <- D.readCsv "./data/housing.csv"-    print $ df-        |> D.derive "rooms_per_household" (total_rooms / households)-        |> D.filterWhere (median_income .>. 5)-        |> D.groupBy ["ocean_proximity"]-        |> D.aggregate [F.mean median_house_value `as` "avg_value"]+df <- D.readCsv "./data/housing.csv"+df |> D.derive "rooms_per_household" (total_rooms / households)+   |> D.filterWhere (median_income .>. 5)+   |> D.groupBy ["ocean_proximity"]+   |> D.aggregate [F.mean median_house_value `as` "avg_value"]+   |> D.toMarkdown' ``` +> <!-- sabela:mime text/plain -->+> | ocean_proximity<br>Text | avg_value<br>Double |+> | ------------------------|-------------------- |+> | NEAR BAY                | 361441.9354304636   |+> | NEAR OCEAN              | 380041.63071895426  |+> | INLAND                  | 234817.86695906433  |+> | <1H OCEAN               | 333411.75125531096  |++ Compare this to the manual version which requires spelling out every column name and type: + ```haskell -- Without TH — every column needs its name and type spelled out df |> D.derive "rooms_per_household"         (F.col @Double "total_rooms" / F.col @Double "households")    |> D.filterWhere (F.col @Double "median_income" .>. F.lit 5)+   |> D.take 5+   |> D.toMarkdown' ``` +> <!-- sabela:mime text/plain -->+> | longitude<br>Double | latitude<br>Double | housing_median_age<br>Double | total_rooms<br>Double | total_bedrooms<br>Maybe Double | population<br>Double | households<br>Double | median_income<br>Double | median_house_value<br>Double | ocean_proximity<br>Text | rooms_per_household<br>Double |+> | --------------------|--------------------|------------------------------|-----------------------|--------------------------------|----------------------|----------------------|-------------------------|------------------------------|-------------------------|------------------------------ |+> | -122.23             | 37.88              | 41.0                         | 880.0                 | Just 129.0                     | 322.0                | 126.0                | 8.3252                  | 452600.0                     | NEAR BAY                | 6.984126984126984             |+> | -122.22             | 37.86              | 21.0                         | 7099.0                | Just 1106.0                    | 2401.0               | 1138.0               | 8.3014                  | 358500.0                     | NEAR BAY                | 6.238137082601054             |+> | -122.24             | 37.85              | 52.0                         | 1467.0                | Just 190.0                     | 496.0                | 177.0                | 7.2574                  | 352100.0                     | NEAR BAY                | 8.288135593220339             |+> | -122.25             | 37.85              | 52.0                         | 1274.0                | Just 235.0                     | 558.0                | 219.0                | 5.6431000000000004      | 341300.0                     | NEAR BAY                | 5.8173515981735155            |+> | -122.29             | 37.82              | 49.0                         | 135.0                 | Just 29.0                      | 86.0                 | 23.0                 | 6.1183                  | 75000.0                      | NEAR BAY                | 5.869565217391305             |++ ### Generate a schema type from a CSV  `deriveSchemaFromCsvFile` generates a type synonym for use with the typed API — instead of manually writing out every column name and type: -```haskell-{-# LANGUAGE TemplateHaskell #-}-{-# LANGUAGE DataKinds #-} -import qualified DataFrame.Typed as T-+```haskell -- Generates:--- type HousingSchema = '[ T.Column "longitude" Double---                        , T.Column "latitude" Double---                        , T.Column "total_rooms" Double---                        , ...---                        ]-$(T.deriveSchemaFromCsvFile "HousingSchema" "./data/housing.csv")+-- type HousingSchema = '[ DT.Column "longitude" Double+--                       , DT.Column "latitude" Double+--                       , DT.Column "total_rooms" Double+--                       , ...+--                       ]+$(DT.deriveSchemaFromCsvFile "HousingSchema" "./data/housing.csv") ``` +> <!-- sabela:mime text/plain -->++ ### Generate a schema (and a row bridge) from a record ADT  When the canonical row shape lives in your code as a Haskell record,@@ -260,27 +306,18 @@ instance that converts between `[Order]` and a `DataFrame` (or `TypedDataFrame OrderSchema`) at runtime: -```haskell-{-# LANGUAGE DataKinds #-}-{-# LANGUAGE FlexibleInstances #-}-{-# LANGUAGE TemplateHaskell #-}-{-# LANGUAGE TypeFamilies #-} -import Data.Int (Int64)-import qualified Data.Text as T-import qualified DataFrame as D-import qualified DataFrame.Typed as DT-+```haskell data Order = Order     { orderId :: Int64-    , region  :: T.Text+    , region  :: Text     , amount  :: Double     } deriving (Show, Eq)  $(DT.deriveSchemaFromType ''Order) -- expands to: --   type OrderSchema =---     '[DT.Column "order_id" Int64, DT.Column "region" T.Text, DT.Column "amount" Double]+--     '[DT.Column "order_id" Int64, DT.Column "region" Text, DT.Column "amount" Double] --   instance DT.HasSchema Order where --     type Schema Order = OrderSchema --     toColumns   = ...@@ -289,18 +326,45 @@ xs :: [Order] xs = [Order 1 "us" 10.0, Order 2 "eu" 20.5] --- Untyped: [Order] <-> DataFrame-df :: D.DataFrame-df = D.fromRecords xs+-- Untyped: [Order] -> DataFrame+ordersDf :: D.DataFrame+ordersDf = D.fromRecords xs -xs' :: Either T.Text [Order]-xs' = D.toRecords df          -- runtime-checked+ordersDf |> D.toMarkdown'+``` --- Typed: [Order] <-> TypedDataFrame OrderSchema-tdf :: DT.TypedDataFrame OrderSchema-tdf = DT.fromRecordsTyped xs+> <!-- sabela:mime text/plain -->+> | order_id<br>Int64 | region<br>Text | amount<br>Double |+> | ------------------|----------------|----------------- |+> | 1                 | us             | 10.0             |+> | 2                 | eu             | 20.5             |+++The runtime-checked round-trip back to records:+++```haskell+D.toRecords ordersDf :: Either Text [Order] ``` +> <!-- sabela:mime text/plain -->+> Right [Order {orderId = 1, region = "us", amount = 10.0},Order {orderId = 2, region = "eu", amount = 20.5}]+++And the typed bridge — `[Order]` to `TypedDataFrame OrderSchema` and back:+++```haskell+DT.thaw (DT.fromRecordsTyped xs :: DT.TypedDataFrame OrderSchema) |> D.toMarkdown'+```++> <!-- sabela:mime text/plain -->+> | order_id<br>Int64 | region<br>Text | amount<br>Double |+> | ------------------|----------------|----------------- |+> | 1                 | us             | 10.0             |+> | 2                 | eu             | 20.5             |++ Field names are translated `camelCase → snake_case` by default; override the translation with `deriveSchemaFromTypeWith defaultSchemaOptions{nameTransform = id}` (or any `String -> String`).@@ -309,6 +373,7 @@ typed-dataframe machinery), there's a companion splice in `DataFrame.Internal.Schema` (re-exported from `DataFrame`): + ```haskell $(D.deriveSchema ''Order) -- emits:@@ -323,92 +388,114 @@  orders :: IO D.DataFrame orders = do-    df <- D.readCsvWithSchema orderSchema "orders.csv"-    pure (D.filter orderAmount (> 100) df)+    raw <- D.readCsvWithSchema orderSchema "./data/orders.csv"+    pure (D.filter orderAmount (> 100) raw) ``` +> <!-- sabela:mime text/plain -->++ Each record field gets a typed accessor named `<lower-first TyConName><UpperFirst FieldName>`, so `data Order { customerId :: Int }` yields `orderCustomerId :: Expr Int = col "customer_id"`. That's the same shape as `$(D.declareColumns df)` produces from a runtime `DataFrame`, but driven off the ADT instead of an existing frame.  If you'd rather not depend on Template Haskell, the same schema is-available via `GHC.Generics`:+available via `GHC.Generics` (shown here on an equivalent record): -```haskell-{-# LANGUAGE DeriveGeneric #-}-{-# LANGUAGE TypeFamilies #-}-{-# LANGUAGE UndecidableInstances #-} +```haskell import GHC.Generics (Generic) import DataFrame.Typed (Schema)-import qualified DataFrame.Typed as DT -data Order = Order { … } deriving (Generic)+data OrderG = OrderG+    { orderGId :: Int64+    , regionG  :: Text+    , amountG  :: Double+    } deriving (Generic) -type OrderSchema = DT.SchemaOf Order+type OrderGSchema = DT.SchemaOf OrderG -instance DT.HasSchema Order where-    type Schema Order = OrderSchema+instance DT.HasSchema OrderG where+    type Schema OrderG = OrderGSchema     toColumns   = DT.genericToColumns     fromColumns = DT.genericFromColumns ``` +> <!-- sabela:mime text/plain -->++ ## Typed API  When you want compile-time guarantees that column names exist and types match, wrap your `DataFrame` in a `TypedDataFrame`: -```haskell-{-# LANGUAGE DataKinds #-}-{-# LANGUAGE TypeApplications #-}-{-# LANGUAGE OverloadedStrings #-} -import qualified DataFrame as D-import qualified DataFrame.Typed as T-import Data.Text (Text)-import DataFrame.Operators-+```haskell type EmployeeSchema =-    '[ T.Column "name"       Text-     , T.Column "department" Text-     , T.Column "salary"     Double+    '[ DT.Column "name"       Text+     , DT.Column "department" Text+     , DT.Column "salary"     Double      ] -main :: IO ()-main = do-    df <- D.readCsv "employees.csv"-    case T.freeze @EmployeeSchema df of-        Nothing  -> putStrLn "Schema mismatch!"-        Just tdf -> do-            let result = tdf-                    |> T.derive @"bonus" (T.col @"salary" * T.lit 0.1)-                    |> T.filterWhere (T.col @"salary" .>. T.lit 50000)-                    |> T.select @'["name", "bonus"]-            print (T.thaw result)+employees <- D.readCsv "./data/employees.csv"+case DT.freeze @EmployeeSchema employees of+    Nothing  -> "Schema mismatch!"+    Just tdf -> tdf+        |> DT.derive @"bonus" (DT.col @"salary" * DT.lit 0.1)+        |> DT.filterWhere (DT.col @"salary" DT..>. DT.lit 50000)+        |> DT.select @'["name", "bonus"]+        |> DT.thaw+        |> D.toMarkdown' ``` -`T.freeze` validates the runtime `DataFrame` against your schema once at the boundary. After that, every column access is checked at compile time:+> <!-- sabela:mime text/plain -->+> | name<br>Text | bonus<br>Double |+> | -------------|---------------- |+> | Alice        | 8500.0          |+> | Carol        | 12000.0         |+> | Dave         | 5200.0          |+> | Frank        | 6700.0          | -```haskell--- Typo in column name → compile error-tdf |> T.filterWhere (T.col @"slary" .>. T.lit 50000)++`DT.freeze` validates the runtime `DataFrame` against your schema once at the boundary. After that, every column access is checked at compile time:+++```text+-- Typo in column name -> compile error+tdf |> DT.filterWhere (DT.col @"slary" DT..>. DT.lit 50000) -- error: Column "slary" not found in schema --- Wrong type → compile error-tdf |> T.filterWhere (T.col @"name" .>. T.lit 50000)+-- Wrong type -> compile error+tdf |> DT.filterWhere (DT.col @"name" DT..>. DT.lit 50000) -- error: Couldn't match type 'Text' with 'Double' ``` + `filterAllJust` goes further — it strips `Maybe` from every column in the schema type, so downstream code can't accidentally treat cleaned columns as nullable: + ```haskell--- Before: TypedDataFrame '[Column "score" (Maybe Double), Column "name" Text]-let cleaned = T.filterAllJust tdf--- After:  TypedDataFrame '[Column "score" Double, Column "name" Text]+type ScoreSchema = '[ DT.Column "name" Text, DT.Column "score" (Maybe Double) ] -cleaned |> T.derive @"scaled" (T.col @"score" * T.lit 100)+scoresDf = D.fromNamedColumns+    [ ("name",  D.fromList ["a", "b", "c" :: Text])+    , ("score", D.fromList [Just 1.0, Nothing, Just 3.0 :: Maybe Double])+    ]++Just stdf = DT.freeze @ScoreSchema scoresDf++-- filterAllJust drops the null row and changes the column type from+-- (Maybe Double) to Double, so `scaled` can multiply it directly.+DT.thaw (DT.filterAllJust stdf |> DT.derive @"scaled" (DT.col @"score" * DT.lit 100)) |> D.toMarkdown' ``` +> <!-- sabela:mime text/plain -->+> | name<br>Text | score<br>Double | scaled<br>Double |+> | -------------|-----------------|----------------- |+> | a            | 1.0             | 100.0            |+> | c            | 3.0             | 300.0            |++ ## Features  **I/O**: CSV, TSV, Parquet (Snappy, ZSTD, Gzip), JSON. Read Parquet from HTTP URLs and Hugging Face datasets (`hf://` URIs). Column projection and predicate pushdown for Parquet reads.@@ -433,30 +520,49 @@  For files too large to fit in memory, `DataFrame.Lazy` provides a streaming query engine. Declare a schema, build a query plan with the same familiar operations, and `runDataFrame` runs it through an optimizer before streaming results batch-by-batch: + ```haskell import qualified DataFrame.Lazy as L-import qualified DataFrame.Functions as F-import DataFrame.Operators-import DataFrame.Internal.Schema (Schema, schemaType)-import Data.Text (Text)+import DataFrame.Internal.Schema (schemaType, makeSchema) -mySchema :: Schema-mySchema = [ ("name",   schemaType @Text)-           , ("weight", schemaType @Double)-           , ("height", schemaType @Double)-           ]+housingSchema = makeSchema+    [ ("longitude",          schemaType @Double)+    , ("latitude",           schemaType @Double)+    , ("housing_median_age", schemaType @Double)+    , ("total_rooms",        schemaType @Double)+    , ("total_bedrooms",     schemaType @(Maybe Double))+    , ("population",         schemaType @Double)+    , ("households",         schemaType @Double)+    , ("median_income",      schemaType @Double)+    , ("median_house_value", schemaType @Double)+    , ("ocean_proximity",    schemaType @Text)+    ] -main :: IO ()-main = do-    result <- L.runDataFrame $-        L.scanCsv mySchema "large_file.csv"-        |> L.filter  (F.col @Double "height" .>. F.lit 1.7)-        |> L.select  ["name", "weight", "height"]-        |> L.derive  "bmi" (F.col @Double "weight"-                           / (F.col @Double "height" * F.col @Double "height"))-        |> L.take 1000-    print result+lazyResult <- L.runDataFrame $+    L.scanCsv housingSchema "./data/housing.csv"+    |> L.filter  (F.col @Double "median_income" .>. F.lit 5)+    |> L.derive  "value_per_income"+                 (F.col @Double "median_house_value" / F.col @Double "median_income")+    |> L.select  ["ocean_proximity", "median_house_value", "value_per_income"]+    |> L.take 1000++D.take 10 lazyResult |> D.toMarkdown' ```++> <!-- sabela:mime text/plain -->+> | ocean_proximity<br>Text | median_house_value<br>Double | value_per_income<br>Double |+> | ------------------------|------------------------------|--------------------------- |+> | NEAR BAY                | 452600.0                     | 54365.06029885168          |+> | NEAR BAY                | 358500.0                     | 43185.48678536151          |+> | NEAR BAY                | 352100.0                     | 48515.997464656764         |+> | NEAR BAY                | 341300.0                     | 60480.94132657581          |+> | NEAR BAY                | 75000.0                      | 12258.307046074891         |+> | NEAR BAY                | 262500.0                     | 51554.49064163246          |+> | NEAR BAY                | 327600.0                     | 55908.25312308007          |+> | NEAR BAY                | 347600.0                     | 65748.65703260951          |+> | NEAR BAY                | 366100.0                     | 61467.42780389524          |+> | NEAR BAY                | 373600.0                     | 58895.860264211624         |+  The optimizer pushes the filter into the scan, drops unreferenced columns before reading, and stops pulling batches once 1000 rows have been collected. 

@@ -1,16 +1,27 @@ {-# LANGUAGE NumericUnderscores #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE TypeApplications #-}+{-# OPTIONS_GHC -fno-warn-orphans #-}  import qualified DataFrame as D import qualified DataFrame.Functions as F +import Control.DeepSeq (NFData (..)) import Control.Monad (void) import Criterion.Main+import DataFrame.Internal.DataFrame (forceDataFrame) import DataFrame.Operations.Join import DataFrame.Operators import System.Process hiding (env) import System.Random.Stateful++{- | Criterion's 'nf' and 'env' force benchmark inputs/results to normal form+via 'NFData'. The core library intentionally dropped its @instance NFData+DataFrame@ in favour of 'forceDataFrame', so we provide a thin orphan here+(scoped to the benchmark) that reuses it.+-}+instance NFData D.DataFrame where+    rnf df = forceDataFrame df `seq` ()  haskell :: IO () haskell = do

@@ -1,6 +1,6 @@ cabal-version:      3.0 name:               dataframe-version:            2.1.0.0+version:            2.1.0.1  synopsis: A fast, safe, and intuitive DataFrame library. @@ -192,7 +192,6 @@         buildable: False     build-depends:         base        >= 4   && < 5,-        dataframe,         dataframe-core ^>= 1.0,         dataframe-csv ^>= 1.0,         dataframe-json ^>= 1.0,@@ -278,6 +277,7 @@     hs-source-dirs: benchmark     build-depends: base >= 4 && < 5,                    criterion >= 1 && < 2,+                   deepseq >= 1.4 && < 2,                    process >= 1.6 && < 2,                    dataframe >= 1 && < 3,                    random >= 1 && < 2,@@ -339,7 +339,6 @@                     dataframe-operations ^>= 1.0,                     dataframe-parquet ^>= 1.0,                     dataframe-parsing ^>= 1.0,-                    dataframe-th ^>= 1.0,                     HUnit ^>= 1.6,                     QuickCheck >= 2 && < 3,                     random-shuffle >= 0.0.4 && < 1,