diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,8 @@ # Revision history for dataframe +## 0.3.3.2 +* Update documentation on both readthedocs and hackage. + ## 0.3.3.1 * Fix bug in `randomSplit` causing two splits to overlap. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -31,98 +31,4 @@ * Designed for interactivity: expressive syntax, helpful error messages, and sensible defaults. * Works seamlessly in both command-line and notebook environments—great for exploration and scripting alike. -## Example usage - -### Interactive environment -![Screencast of usage in GHCI](./static/example.gif) - -Key features in example: -* Intuitive, SQL-like API to get from data to insights. -* Create typed, completion-ready references to columns in a dataframe using `:exposeColumns` -* Type-safe column transformations for faster and safer exploration. -* Fluid, chaining API that makes code easy to reason about. - -### Standalone script example -```haskell --- Useful Haskell extensions. -{-# LANGUAGE OverloadedStrings #-} -- Allow string literal to be interpreted as any other string type. -{-# LANGUAGE TypeApplications #-} -- Convenience syntax for specifiying the type `sum a b :: Int` vs `sum @Int a b'. - -import qualified DataFrame as D -- import for general functionality. -import qualified DataFrame.Functions as F -- import for column expressions. - -import DataFrame ((|>)) -- import chaining operator with unqualified. - -main :: IO () -main = do - df <- D.readTsv "./data/chipotle.tsv" - let quantity = F.col "quantity" :: D.Expr Int -- A typed reference to a column. - print (df - |> D.select ["item_name", "quantity"] - |> D.groupBy ["item_name"] - |> D.aggregate [ (F.sum quantity) `F.as` "sum_quantity" - , (F.mean quantity) `F.as` "mean_quantity" - , (F.maximum quantity) `F.as` "maximum_quantity" - ] - |> D.sortBy D.Descending ["sum_quantity"] - |> D.take 10) - -``` - -Output: - -``` ------------------------------------------------------------------------------------------- -index | item_name | sum_quantity | mean_quanity | maximum_quanity -------|------------------------------|--------------|--------------------|---------------- - Int | Text | Int | Double | Int -------|------------------------------|--------------|--------------------|---------------- -0 | Chicken Bowl | 761 | 1.0482093663911847 | 3 -1 | Chicken Burrito | 591 | 1.0687160940325497 | 4 -2 | Chips and Guacamole | 506 | 1.0563674321503131 | 4 -3 | Steak Burrito | 386 | 1.048913043478261 | 3 -4 | Canned Soft Drink | 351 | 1.1661129568106312 | 4 -5 | Chips | 230 | 1.0900473933649288 | 3 -6 | Steak Bowl | 221 | 1.04739336492891 | 3 -7 | Bottled Water | 211 | 1.3024691358024691 | 10 -8 | Chips and Fresh Tomato Salsa | 130 | 1.1818181818181819 | 15 -9 | Canned Soda | 126 | 1.2115384615384615 | 4 -``` - -Full example in `./examples` folder using many of the constructs in the API. - -## Installing - -### Jupyter notebook -* We have a [hosted version of the Jupyter notebook](https://ulwazi-exh9dbh2exbzgbc9.westus-01.azurewebsites.net/lab) on azure sites. This is hosted on Azure's free tier so it can only support 3 or 4 kernels at a time. -* To get started quickly, use the Dockerfile in the [ihaskell-dataframe](https://github.com/mchav/ihaskell-dataframe) to build and run an image with dataframe integration. -* For a preview check out the [California Housing](https://github.com/mchav/dataframe/blob/main/docs/California%20Housing.ipynb) notebook. - -### CLI -* Run the installation script `curl '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/mchav/dataframe/refs/heads/main/scripts/install.sh | sh` -* Download the run script with: `curl --output dataframe "https://raw.githubusercontent.com/mchav/dataframe/refs/heads/main/scripts/dataframe.sh"` -* Make the script executable: `chmod +x dataframe` -* Add the script your path: `export PATH=$PATH:./dataframe` -* Run the script with: `dataframe` - - -## What is exploratory data analysis? -We provide a primer [here](https://github.com/mchav/dataframe/blob/main/docs/exploratory_data_analysis_primer.md) and show how to do some common analyses. - -## Coming from other dataframe libraries -Familiar with another dataframe library? Get started: -* [Coming from Pandas](https://github.com/mchav/dataframe/blob/main/docs/coming_from_pandas.md) -* [Coming from Polars](https://github.com/mchav/dataframe/blob/main/docs/coming_from_polars.md) -* [Coming from dplyr](https://github.com/mchav/dataframe/blob/main/docs/coming_from_dplyr.md) - -## Supported input formats -* CSV -* Apache Parquet - -## Supported output formats -* CSV - -## Future work -* Apache arrow compatability -* Integration with more data formats (SQLite, Postgres, json lines, xlsx). -* Host the whole library + Jupyter lab on Azure with auth and isolation. +For an installation guide and tutorials checkout the [project documentation](https://dataframe.readthedocs.io/) and for an API reference checkout the [hackage documentation](https://hackage-content.haskell.org/package/dataframe-0.3.3.2/docs/DataFrame.html). diff --git a/dataframe.cabal b/dataframe.cabal --- a/dataframe.cabal +++ b/dataframe.cabal @@ -1,6 +1,6 @@ cabal-version: 2.4 name: dataframe -version: 0.3.3.1 +version: 0.3.3.2 synopsis: A fast, safe, and intuitive DataFrame library. diff --git a/src/DataFrame.hs b/src/DataFrame.hs --- a/src/DataFrame.hs +++ b/src/DataFrame.hs @@ -18,10 +18,32 @@ Example session: +We provide a script that imports the core functionality and defines helpful +macros for writing safe code. + @ --- GHCi quality-of-life: -ghci> :set -XOverloadedStrings -XTypeApplications -ghci> :module + DataFrame as D, DataFrame.Functions as F, Data.Text (Text) +\$ curl --output dataframe \"https:\/\/raw.githubusercontent.com\/mchav\/dataframe\/refs\/heads\/main\/scripts\/dataframe.sh\" +\$ chmod +x dataframe +\$ export PATH=$PATH:$PWD/dataframe +\$ dataframe +Configuring library for fake-package-0... +Warning: No exposed modules +GHCi, version 9.6.7: https:\/\/www.haskell.org\/ghc\/ :? for help +Loaded GHCi configuration from \/tmp\/cabal-repl.-242816\/setcwd.ghci +======================================== + 📦Dataframe +======================================== + +✨ Modules were automatically imported. + +💡 Use prefix 'D' for core functionality. + ● E.g. D.readCsv \"\/path\/to\/file\" +💡 Use prefix 'F' for expression functions. + ● E.g. F.sum (F.col \@Int \"value\") + +✅ Ready. +Loaded GHCi configuration from ./dataframe.ghci +ghci> @ = Quick start @@ -48,17 +70,18 @@ 9 | longitude | 20640 | 0 | 0 | 844 | Double -- 2) Project & filter -ghci> df1 = D.filter \@Text "ocean_proximity" (== \"ISLAND\") df0 D.|> D.select ["median_house_value", "median_income", "ocean_proximity"] +ghci> :exposeColumn df +ghci> df1 = D.filterWhere (ocean_proximity F.== F.lit \"ISLAND\") df0 D.|> D.select [F.name median_house_value, F.name median_income, F.name ocean_proximity] -- 3) Add a derived column using the expression DSL -- (col types are explicit via TypeApplications) -ghci> df2 = D.derive "rooms_per_household" (F.col \@Double "total_rooms" / F.col \@Double "households") df0 +ghci> df2 = D.derive "rooms_per_household" (total_rooms / households) df0 -- 4) Group + aggregate ghci> let grouped = D.groupBy ["ocean_proximity"] df0 ghci> let summary = D.aggregate - [ F.maximum (F.col \@Double "median_house_value") \`F.as\` "max_house_value"] + [ F.maximum median_house_value \`F.as\` "max_house_value"] grouped ghci> D.take 5 summary ----------------------------------------- @@ -92,7 +115,7 @@ __Row ops__ - * @D.filter :: Columnable a => Text -> (a -> Bool) -> DataFrame -> DataFrame@ + * @D.filterWhere :: Expr Bool -> DataFrame -> DataFrame@ * @D.sortBy :: SortOrder -> [Text] -> DataFrame -> DataFrame@ __Column ops__ diff --git a/src/DataFrame/Operations/Subset.hs b/src/DataFrame/Operations/Subset.hs --- a/src/DataFrame/Operations/Subset.hs +++ b/src/DataFrame/Operations/Subset.hs @@ -156,10 +156,9 @@ filterBy :: (Columnable a) => (a -> Bool) -> T.Text -> DataFrame -> DataFrame filterBy = flip filter -{- | O(k) filters the dataframe with a row predicate. The arguments in the function - must appear in the same order as they do in the list. +{- | O(k) filters the dataframe with a boolean expression. -> filterWhere (["x", "y"], func (\x y -> x + y > 5)) df +> filterWhere (F.col @Int x + F.col y F.> 5) df -} filterWhere :: Expr Bool -> DataFrame -> DataFrame filterWhere expr df =