dataframe 0.3.4.1 → 0.3.5.0

raw patch · 80 files changed

+319/−274 lines, 80 filesdep ~granitedep ~randombinary-addedPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

Dependency ranges changed: granite, random

API changes (from Hackage documentation)

Files

• CHANGELOG.md +50/−0
• app/Benchmark.hs +25/−11
• dataframe.cabal +6/−5
• src/DataFrame/Display/Terminal/Plot.hs +15/−16
• src/DataFrame/Display/Web/Plot.hs +15/−16
• src/DataFrame/Functions.hs +12/−5
• src/DataFrame/IO/Parquet.hs +2/−1
• src/DataFrame/IO/Parquet/Page.hs +8/−2
• src/DataFrame/IO/Parquet/Thrift.hs +20/−13
• src/DataFrame/Internal/Expression.hs +81/−191
• src/DataFrame/Monad.hs +50/−0
• src/DataFrame/Operations/Transformations.hs +15/−1
• src/DataFrame/Synthesis.hs +1/−1
• tests/Functions.hs +17/−10
• tests/Parquet.hs +2/−2
• tests/data/alltypes_dictionary.parquet binary
• tests/data/alltypes_plain.parquet binary
• tests/data/alltypes_plain.snappy.parquet binary
• tests/data/alltypes_tiny_pages.parquet binary
• tests/data/alltypes_tiny_pages_plain.parquet binary
• tests/data/binary.parquet binary
• tests/data/binary_truncated_min_max.parquet binary
• tests/data/byte_array_decimal.parquet binary
• tests/data/byte_stream_split.zstd.parquet binary
• tests/data/byte_stream_split_extended.gzip.parquet binary
• tests/data/column_chunk_key_value_metadata.parquet binary
• tests/data/concatenated_gzip_members.parquet binary
• tests/data/data_index_bloom_encoding_stats.parquet binary
• tests/data/data_index_bloom_encoding_with_length.parquet binary
• tests/data/datapage_v1-corrupt-checksum.parquet binary
• tests/data/datapage_v1-snappy-compressed-checksum.parquet binary
• tests/data/datapage_v1-uncompressed-checksum.parquet binary
• tests/data/datapage_v2.snappy.parquet binary
• tests/data/datapage_v2_empty_datapage.snappy.parquet binary
• tests/data/delta_binary_packed.parquet binary
• tests/data/delta_byte_array.parquet binary
• tests/data/delta_encoding_optional_column.parquet binary
• tests/data/delta_encoding_required_column.parquet binary
• tests/data/delta_length_byte_array.parquet binary
• tests/data/dict-page-offset-zero.parquet binary
• tests/data/fixed_length_byte_array.parquet binary
• tests/data/fixed_length_decimal.parquet binary
• tests/data/fixed_length_decimal_legacy.parquet binary
• tests/data/float16_nonzeros_and_nans.parquet binary
• tests/data/float16_zeros_and_nans.parquet binary
• tests/data/hadoop_lz4_compressed.parquet binary
• tests/data/hadoop_lz4_compressed_larger.parquet binary
• tests/data/incorrect_map_schema.parquet binary
• tests/data/int32_decimal.parquet binary
• tests/data/int32_with_null_pages.parquet binary
• tests/data/int64_decimal.parquet binary
• tests/data/int96_from_spark.parquet binary
• tests/data/large_string_map.brotli.parquet binary
• tests/data/list_columns.parquet binary
• tests/data/lz4_raw_compressed.parquet binary
• tests/data/lz4_raw_compressed_larger.parquet binary
• tests/data/map_no_value.parquet binary
• tests/data/mtcars.parquet binary
• tests/data/nan_in_stats.parquet binary
• tests/data/nation.dict-malformed.parquet binary
• tests/data/nested_lists.snappy.parquet binary
• tests/data/nested_maps.snappy.parquet binary
• tests/data/nested_structs.rust.parquet binary
• tests/data/non_hadoop_lz4_compressed.parquet binary
• tests/data/nonnullable.impala.parquet binary
• tests/data/null_list.parquet binary
• tests/data/nullable.impala.parquet binary
• tests/data/nulls.snappy.parquet binary
• tests/data/old_list_structure.parquet binary
• tests/data/overflow_i16_page_cnt.parquet binary
• tests/data/page_v2_empty_compressed.parquet binary
• tests/data/plain-dict-uncompressed-checksum.parquet binary
• tests/data/repeated_no_annotation.parquet binary
• tests/data/repeated_primitive_no_list.parquet binary
• tests/data/rle-dict-snappy-checksum.parquet binary
• tests/data/rle-dict-uncompressed-corrupt-checksum.parquet binary
• tests/data/rle_boolean_encoding.parquet binary
• tests/data/single_nan.parquet binary
• tests/data/sort_columns.parquet binary
• tests/data/unknown-logical-type.parquet binary

@@ -1,5 +1,55 @@ # Revision history for dataframe +## 0.3.5.0+* Add a `deriveWithExpr` that returns an expression that you can use in a subsequent expressions.+* Add `declareColumnsFromCsvFile` which can create the expressions up front for use in scripts.+    ```haskell+    import qualified DataFrame as D+    import qualified DataFrame.Functions as F++    import Data.Text (Text)+    import DataFrame.Functions ((.==), (.>=))++    $(F.declareColumnsFromCsvFile "./data/housing.csv")++    main :: IO ()+    main = do+        df <- D.readCsv "./data/housing.csv"+        let (df', test) = D.deriveWithExpr "test" (median_house_value .>= 500000) df+        print (D.filterWhere test df')+    ```+* Fix bounds on random.+* Parquet Column chunks weren't reading properly because we didn't correctly calculate the list size.+* Sum function had a bug where the first number was summed twice.+* Add monadic interface for building dataframe expressions that makes schema evolution nice.+    ```haskell+    {-# LANGUAGE OverloadedStrings #-}+    {-# LANGUAGE TemplateHaskell #-}++    module Main where++    import qualified DataFrame as D+    import qualified DataFrame.Functions as F++    import DataFrame.Monad++    import Data.Text (Text)+    import DataFrame.Functions ((.&&), (.>=))++    $(F.declareColumnsFromCsvFile "./data/housing.csv")++    main :: IO ()+    main = do+        df <- D.readCsv "./data/housing.csv"+        print $ runFrameM df $ do+            is_expensive <- deriveM "is_expensive" (median_house_value .>= 500000)+            filterWhereM is_expensive+            luxury <- deriveM "luxury" (is_expensive .&& median_income .>= 8)+            filterWhereM luxury+    ```+* Change order of exponentiation to putting the exponent second. It was initially first cause of some internal efficiency detail but that's silly.+* Fix bug where we didn't concat columns from row groups.+ ## 0.3.4.1 * Faster sum operation (now does a reduction instead of collecting the vector and aggregating) * Update the fixity of comparison operations. Before `(x + y) .<= 10`. Now: `x + y ,<= 10`.

@@ -8,17 +8,31 @@ import qualified DataFrame.Functions as F import System.Random.Stateful -import DataFrame ((|>))- main :: IO () main = do-    df <- D.readCsv "../db-benchmark/data/G1_2e6_1e2_0_0.csv"+    let n = 100_000_000+    g <- newIOGenM =<< newStdGen+    let range = (0 :: Double, 1 :: Double)+    startGeneration <- getCurrentTime+    ns <- VU.replicateM n (uniformRM range g)+    xs <- VU.replicateM n (uniformRM range g)+    ys <- VU.replicateM n (uniformRM range g)+    let df = D.fromUnnamedColumns (map D.fromUnboxedVector [ns, xs, ys])     print df-    start <- getCurrentTime-    print $-        df-            |> D.groupBy ["id1"]-            |> D.aggregate [F.sum (F.col @Int "v1") `F.as` "v1_sum"]-    end <- getCurrentTime-    let computeTime = diffUTCTime end start-    putStrLn $ "Compute Time: " ++ show computeTime+    endGeneration <- getCurrentTime+    let generationTime = diffUTCTime endGeneration startGeneration+    putStrLn $ "Data generation Time: " ++ show generationTime+    startCalculation <- getCurrentTime+    print $ D.mean (F.col @Double "0") df+    print $ D.variance (F.col @Double "1") df+    print $ D.correlation "1" "2" df+    endCalculation <- getCurrentTime+    let calculationTime = diffUTCTime endCalculation startCalculation+    putStrLn $ "Calculation Time: " ++ show calculationTime+    startFilter <- getCurrentTime+    print $ D.filter (F.col @Double "0") (> 0.971) df D.|> D.take 10+    endFilter <- getCurrentTime+    let filterTime = diffUTCTime endFilter startFilter+    putStrLn $ "Filter Time: " ++ show filterTime+    let totalTime = diffUTCTime endFilter startGeneration+    putStrLn $ "Total Time: " ++ show totalTime

@@ -1,6 +1,6 @@ cabal-version:      2.4 name:               dataframe-version:            0.3.4.1+version:            0.3.5.0  synopsis: A fast, safe, and intuitive DataFrame library. @@ -18,7 +18,7 @@ extra-doc-files: CHANGELOG.md README.md extra-source-files: cbits/process_csv.h                     tests/data/*.csv-                    -- tests/data/*.parquet+                    tests/data/*.parquet                     -- tests/data/*.md                     -- tests/data/*.bin                     -- tests/data/*.json@@ -83,7 +83,8 @@                     DataFrame.IO.Parquet.Time,                     DataFrame.IO.Parquet.Types,                     DataFrame.Lazy.IO.CSV,-                    DataFrame.Lazy.Internal.DataFrame+                    DataFrame.Lazy.Internal.DataFrame,+                    DataFrame.Monad     build-depends:    base >= 4 && <5,                       aeson >= 0.11.0.0 && < 3,                       array >= 0.5.4.0 && < 0.6,@@ -93,11 +94,11 @@                       cassava >= 0.1 && < 1,                       containers >= 0.6.7 && < 0.9,                       directory >= 1.3.0.0 && < 2,-                      granite ^>= 0.3,+                      granite == 0.3.0.5,                       hashable >= 1.2 && < 2,                       process ^>= 1.6,                       snappy-hs ^>= 0.1,-                      random >= 1 && < 2,+                      random >= 1.3 && < 2,                       regex-tdfa >= 1.3.0 && < 2,                       scientific >=0.3.1 && <0.4,                       template-haskell >= 2.0 && < 3,

@@ -1,5 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes #-} {-# LANGUAGE ExplicitNamespaces #-}+{-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RankNTypes #-}@@ -14,14 +15,14 @@ import qualified Data.Text as T import qualified Data.Text.IO as T import Data.Type.Equality (TestEquality (testEquality), type (:~:) (Refl))-import Data.Typeable (Typeable) import qualified Data.Vector as V import qualified Data.Vector.Generic as VG import qualified Data.Vector.Unboxed as VU+import DataFrame.Internal.Types import GHC.Stack (HasCallStack) import Type.Reflection (typeRep) -import DataFrame.Internal.Column (Column (..), isNumeric)+import DataFrame.Internal.Column (Column (..), Columnable, isNumeric) import qualified DataFrame.Internal.Column as D import DataFrame.Internal.DataFrame (DataFrame (..), getColumn) import DataFrame.Internal.Expression@@ -379,28 +380,26 @@                     UnboxedColumn vec -> unboxedVectorToDoubles vec                     _ -> [] -vectorToDoubles :: forall a. (Typeable a, Show a) => V.Vector a -> [Double]+vectorToDoubles :: forall a. (Columnable a, Show a) => V.Vector a -> [Double] vectorToDoubles vec =     case testEquality (typeRep @a) (typeRep @Double) of         Just Refl -> V.toList vec-        Nothing -> case testEquality (typeRep @a) (typeRep @Int) of-            Just Refl -> V.toList $ V.map fromIntegral vec-            Nothing -> case testEquality (typeRep @a) (typeRep @Integer) of-                Just Refl -> V.toList $ V.map fromIntegral vec-                Nothing -> case testEquality (typeRep @a) (typeRep @Float) of-                    Just Refl -> V.toList $ V.map realToFrac vec-                    Nothing -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"+        Nothing -> case sIntegral @a of+            STrue -> V.toList $ V.map fromIntegral vec+            SFalse -> case sFloating @a of+                STrue -> V.toList $ V.map realToFrac vec+                SFalse -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"  unboxedVectorToDoubles ::-    forall a. (Typeable a, VU.Unbox a, Show a) => VU.Vector a -> [Double]+    forall a. (Columnable a, VU.Unbox a, Show a) => VU.Vector a -> [Double] unboxedVectorToDoubles vec =     case testEquality (typeRep @a) (typeRep @Double) of         Just Refl -> VU.toList vec-        Nothing -> case testEquality (typeRep @a) (typeRep @Int) of-            Just Refl -> VU.toList $ VU.map fromIntegral vec-            Nothing -> case testEquality (typeRep @a) (typeRep @Float) of-                Just Refl -> VU.toList $ VU.map realToFrac vec-                Nothing -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"+        Nothing -> case sIntegral @a of+            STrue -> VU.toList $ VU.map fromIntegral vec+            SFalse -> case sFloating @a of+                STrue -> VU.toList $ VU.map realToFrac vec+                SFalse -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"  groupWithOther :: Int -> [(T.Text, Double)] -> [(T.Text, Double)] groupWithOther n items =

@@ -1,5 +1,6 @@ {-# LANGUAGE AllowAmbiguousTypes #-} {-# LANGUAGE ExplicitNamespaces #-}+{-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs #-} {-# LANGUAGE OverloadedStrings #-} {-# LANGUAGE RankNTypes #-}@@ -15,7 +16,6 @@ import qualified Data.Text as T import qualified Data.Text.IO as T import Data.Type.Equality (TestEquality (testEquality), type (:~:) (Refl))-import Data.Typeable (Typeable) import qualified Data.Vector as V import qualified Data.Vector.Generic as VG import qualified Data.Vector.Unboxed as VU@@ -23,10 +23,11 @@ import System.Random (newStdGen, randomRs) import Type.Reflection (typeRep) -import DataFrame.Internal.Column (Column (..), isNumeric)+import DataFrame.Internal.Column (Column (..), Columnable, isNumeric) import qualified DataFrame.Internal.Column as D import DataFrame.Internal.DataFrame (DataFrame (..), getColumn) import DataFrame.Internal.Expression+import DataFrame.Internal.Types import DataFrame.Operations.Core import qualified DataFrame.Operations.Subset as D import Numeric (showFFloat)@@ -868,28 +869,26 @@                     UnboxedColumn vec -> unboxedVectorToDoubles vec                     _ -> [] -vectorToDoubles :: forall a. (Typeable a, Show a) => V.Vector a -> [Double]+vectorToDoubles :: forall a. (Columnable a, Show a) => V.Vector a -> [Double] vectorToDoubles vec =     case testEquality (typeRep @a) (typeRep @Double) of         Just Refl -> V.toList vec-        Nothing -> case testEquality (typeRep @a) (typeRep @Int) of-            Just Refl -> V.toList $ V.map fromIntegral vec-            Nothing -> case testEquality (typeRep @a) (typeRep @Integer) of-                Just Refl -> V.toList $ V.map fromIntegral vec-                Nothing -> case testEquality (typeRep @a) (typeRep @Float) of-                    Just Refl -> V.toList $ V.map realToFrac vec-                    Nothing -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"+        Nothing -> case sIntegral @a of+            STrue -> V.toList $ V.map fromIntegral vec+            SFalse -> case sFloating @a of+                STrue -> V.toList $ V.map realToFrac vec+                SFalse -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"  unboxedVectorToDoubles ::-    forall a. (Typeable a, VU.Unbox a, Show a) => VU.Vector a -> [Double]+    forall a. (Columnable a, VU.Unbox a, Show a) => VU.Vector a -> [Double] unboxedVectorToDoubles vec =     case testEquality (typeRep @a) (typeRep @Double) of         Just Refl -> VU.toList vec-        Nothing -> case testEquality (typeRep @a) (typeRep @Int) of-            Just Refl -> VU.toList $ VU.map fromIntegral vec-            Nothing -> case testEquality (typeRep @a) (typeRep @Float) of-                Just Refl -> VU.toList $ VU.map realToFrac vec-                Nothing -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"+        Nothing -> case sIntegral @a of+            STrue -> VU.toList $ VU.map fromIntegral vec+            SFalse -> case sFloating @a of+                STrue -> VU.toList $ VU.map realToFrac vec+                SFalse -> error $ "Column is not numeric (type: " ++ show (typeRep @a) ++ ")"  getCategoricalCounts ::     (HasCallStack) => T.Text -> DataFrame -> Maybe [(T.Text, Double)]

@@ -27,6 +27,7 @@ import DataFrame.Internal.Statistics  import Control.Monad+import Control.Monad.IO.Class import qualified Data.Char as Char import Data.Function import Data.Functor@@ -37,6 +38,7 @@ import Data.Time import qualified Data.Vector as V import qualified Data.Vector.Unboxed as VU+import qualified DataFrame.IO.CSV as CSV import Debug.Trace (trace) import Language.Haskell.TH import qualified Language.Haskell.TH.Syntax as TH@@ -156,7 +158,7 @@ maximum expr = AggReduce expr "maximum" Prelude.max  sum :: forall a. (Columnable a, Num a) => Expr a -> Expr a-sum expr = AggReduce expr "sum" (+)+sum expr = AggFold expr "sum" 0 (+)  sumMaybe :: forall a. (Columnable a, Num a) => Expr (Maybe a) -> Expr a sumMaybe expr = AggVector expr "sumMaybe" (P.sum . catMaybes . V.toList)@@ -199,10 +201,10 @@ zScore :: Expr Double -> Expr Double zScore c = (c - mean c) / stddev c -pow :: (Columnable a, Num a) => Int -> Expr a -> Expr a-pow 0 _ = Lit 1-pow 1 expr = expr-pow i expr = UnaryOp ("pow " <> T.pack (show i)) (^ i) expr+pow :: (Columnable a, Num a) => Expr a -> Int -> Expr a+pow _ 0 = Lit 1+pow expr 1 = expr+pow expr i = UnaryOp ("pow " <> T.pack (show i)) (^ i) expr  relu :: (Columnable a, Num a) => Expr a -> Expr a relu = UnaryOp "relu" (Prelude.max 0)@@ -357,6 +359,11 @@  dropFirstAndLast :: [a] -> [a] dropFirstAndLast = reverse . drop 1 . reverse . drop 1++declareColumnsFromCsvFile :: String -> DecsQ+declareColumnsFromCsvFile path = do+    df <- liftIO (CSV.readCsv path)+    declareColumns df  declareColumns :: DataFrame -> DecsQ declareColumns df =

@@ -97,7 +97,7 @@                     primaryEncoding                     maybeTypeLength -            modifyIORef colMap (M.insert colName column)+            modifyIORef colMap (M.insertWith DI.concatColumnsEither colName column)      finalColMap <- readIORef colMap     let orderedColumns =@@ -268,6 +268,7 @@             DictionaryPageHeader{} -> error "processColumnPages: impossible DictionaryPageHeader"             INDEX_PAGE_HEADER -> error "processColumnPages: impossible INDEX_PAGE_HEADER"             PAGE_TYPE_HEADER_UNKNOWN -> error "processColumnPages: impossible PAGE_TYPE_HEADER_UNKNOWN"+    -- This is N^2. We should probably use mutable columns here.     case cols of         [] -> pure $ DI.fromList ([] :: [Maybe Int])         (c : cs) ->

@@ -96,7 +96,7 @@                         (dataPageHeaderV2, rem') = readPageTypeHeader emptyDataPageHeaderV2 rem 0                      in                         readPageHeader (hdr{pageTypeHeader = dataPageHeaderV2}) rem' identifier-                n -> error $ "Unknown page header field" ++ show n+                n -> error $ "Unknown page header field " ++ show n  readPageTypeHeader ::     PageTypeHeader -> [Word8] -> Int16 -> (PageTypeHeader, [Word8])@@ -132,7 +132,13 @@                      in                         readPageTypeHeader                             (hdr{dictionaryPageIsSorted = isSorted == compactBooleanTrue})-                            (drop 1 rem)+                            -- TODO(mchavinda): The bool logic here is a little tricky.+                            -- If the field is a bool then you can get the value+                            -- from the byte (and you don't have to drop a field).+                            -- But in other cases you do.+                            -- This might become a problem later but in the mean+                            -- time I'm not dropping (this assumes this is the common case).+                            rem                             identifier                 n ->                     error $ "readPageTypeHeader: unsupported identifier " ++ show n

@@ -56,6 +56,8 @@     | I16     | I32     | I64+    | I96+    | FLOAT     | DOUBLE     | STRING     | LIST@@ -238,7 +240,7 @@     STRING -> void (readByteString buf pos)     LIST -> skipList buf pos     STRUCT -> skipToStructEnd buf pos-    _ -> return ()+    _ -> error $ "Unknown field type" ++ show fieldType  skipList :: BS.ByteString -> IORef Int -> IO () skipList buf pos = do@@ -469,9 +471,7 @@                     pos                     identifier                     fieldStack-            _ -> do-                skipFieldData elemType buf pos-                readSchemaElement schemaElement buf pos identifier fieldStack+            _ -> error "Uknown schema element"  readRowGroup ::     RowGroup -> BS.ByteString -> IORef Int -> Int16 -> [Int16] -> IO RowGroup@@ -482,10 +482,13 @@         Just (elemType, identifier) -> case identifier of             1 -> do                 sizeAndType <- readAndAdvance pos buf-                let sizeOnly = fromIntegral ((sizeAndType `shiftR` 4) .&. 0x0f) :: Int+                listSize <-+                    if (sizeAndType `shiftR` 4) .&. 0x0f == 15+                        then readVarIntFromBuffer @Int buf pos+                        else return $ fromIntegral ((sizeAndType `shiftR` 4) .&. 0x0f)                 let _elemType = toTType sizeAndType                 columnChunks <--                    replicateM sizeOnly (readColumnChunk emptyColumnChunk buf pos 0 [])+                    replicateM listSize (readColumnChunk emptyColumnChunk buf pos 0 [])                 readRowGroup (r{rowGroupColumns = columnChunks}) buf pos identifier fieldStack             2 -> do                 totalBytes <- readIntFromBuffer @Int64 buf pos@@ -575,7 +578,7 @@                     pos                     identifier                     fieldStack-            _ -> return c+            _ -> error "Unknown column chunk"  readColumnMetadata ::     ColumnMetaData ->@@ -712,7 +715,7 @@                     identifier                     fieldStack             17 -> return $ error "UNIMPLEMENTED"-            _ -> return cm+            _ -> error $ "Unknown column metadata " ++ show identifier  readEncryptionAlgorithm ::     BS.ByteString -> IORef Int -> Int16 -> [Int16] -> IO EncryptionAlgorithm@@ -838,7 +841,7 @@             2 -> do                 v <- readString buf pos                 readKeyValue (kv{value = v}) buf pos identifier fieldStack-            _ -> return kv+            _ -> error "Unknown kv"  readPageEncodingStats ::     PageEncodingStats ->@@ -866,7 +869,7 @@                     pos                     identifier                     []-            _ -> pure pes+            _ -> error "Unknown page encoding stats"  readParquetEncoding ::     BS.ByteString -> IORef Int -> Int16 -> [Int16] -> IO ParquetEncoding@@ -923,7 +926,7 @@                     pos                     identifier                     fieldStack-            _ -> pure cs+            _ -> error "Unknown statistics"  readSizeStatistics ::     SizeStatistics ->@@ -969,18 +972,22 @@                     pos                     identifier                     fieldStack-            _ -> pure ss+            _ -> error "Unknown size statistics"  footerSize :: Int footerSize = 8  toIntegralType :: Int32 -> TType toIntegralType n+    | n == 0 = BOOL     | n == 1 = I32     | n == 2 = I64+    | n == 3 = I96     | n == 4 = DOUBLE+    | n == 5 = DOUBLE     | n == 6 = STRING-    | otherwise = STRING+    | n == 7 = STRING+    | otherwise = error ("Unknown type in schema: " ++ show n)  readLogicalType ::     BS.ByteString -> IORef Int -> Int16 -> [Int16] -> IO LogicalType

@@ -16,6 +16,7 @@ module DataFrame.Internal.Expression where  import Control.Monad.ST (runST)+import Data.Bifunctor import qualified Data.Map as M import Data.Maybe (fromMaybe, isJust) import Data.String@@ -35,12 +36,6 @@ data Expr a where     Col :: (Columnable a) => T.Text -> Expr a     Lit :: (Columnable a) => a -> Expr a-    If ::-        (Columnable a) =>-        Expr Bool ->-        Expr a ->-        Expr a ->-        Expr a     UnaryOp ::         ( Columnable a         , Columnable b@@ -59,6 +54,12 @@         Expr c ->         Expr b ->         Expr a+    If ::+        (Columnable a) =>+        Expr Bool ->+        Expr a ->+        Expr a ->+        Expr a     AggVector ::         ( VG.Vector v b         , Typeable v@@ -75,6 +76,7 @@         T.Text -> -- Operation name         (a -> a -> a) ->         Expr a+    -- TODO(mchav): Numeric reduce might be superfluous since expressions are already type checked.     AggNumericVector ::         ( Columnable a         , Columnable b@@ -105,195 +107,64 @@     forall a.     (Columnable a) =>     DataFrame -> Expr a -> Either DataFrameException (TypedColumn a)-interpret df (Lit value) =-    pure $ TColumn $ fromVector $ V.replicate (fst $ dataframeDimensions df) value-interpret df (Col name) = case getColumn name df of-    Nothing -> Left $ ColumnNotFoundException name "" (M.keys $ columnIndices df)-    Just col -> pure $ TColumn col-interpret df expr@(If cond l r) = case interpret @Bool df cond of-    Left (TypeMismatchException context) ->-        Left $-            TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show cond)-                    }-                )-    Left e -> Left e-    Right (TColumn conditions) -> case interpret @a df l of-        Left (TypeMismatchException context) ->-            Left $-                TypeMismatchException-                    (context{callingFunctionName = Just "interpret", errorColumnName = Just (show l)})-        Left e -> Left e-        Right (TColumn left) -> case interpret @a df r of-            Left (TypeMismatchException context) ->-                Left $-                    TypeMismatchException-                        (context{callingFunctionName = Just "interpret", errorColumnName = Just (show r)})-            Left e -> Left e-            Right (TColumn right) -> case zipWithColumns-                (\(c :: Bool) (l' :: a, r' :: a) -> if c then l' else r')-                conditions-                (zipColumns left right) of-                Left (TypeMismatchException context) ->-                    Left $-                        TypeMismatchException-                            ( context-                                { callingFunctionName = Just "interpret"-                                , errorColumnName = Just (show expr)-                                }-                            )-                Left e -> Left e-                Right res -> pure $ TColumn res-interpret df expr@(UnaryOp _ (f :: c -> d) value) = case interpret @c df value of-    Left (TypeMismatchException context) ->-        Left $-            TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show value)-                    }-                )-    Left e -> Left e-    Right (TColumn value') -> case mapColumn f value' of-        Left (TypeMismatchException context) ->-            Left $-                TypeMismatchException-                    ( context-                        { callingFunctionName = Just "interpret"-                        , errorColumnName = Just (show expr)-                        }-                    )-        Left e -> Left e-        Right res -> pure $ TColumn res-interpret df expr@(BinaryOp _ (f :: c -> d -> e) (Lit left) (Lit right)) =-    pure $-        TColumn $-            fromVector $-                V.replicate (fst $ dataframeDimensions df) (f left right)-interpret df expr@(BinaryOp _ (f :: c -> d -> e) (Lit left) right) = case interpret @d df right of-    Left (TypeMismatchException context) ->-        Left $-            TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show right)-                    }-                )-    Left e -> Left e-    Right (TColumn right') -> case mapColumn (f left) right' of-        Left (TypeMismatchException context) ->-            Left $-                TypeMismatchException-                    ( context-                        { callingFunctionName = Just "interpret"-                        , errorColumnName = Just (show expr)-                        }-                    )-        Left e -> Left e-        Right res -> pure $ TColumn res-interpret df expr@(BinaryOp _ (f :: c -> d -> e) left (Lit right)) = case interpret @c df left of-    Left (TypeMismatchException context) ->-        Left $-            TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show left)-                    }-                )-    Left e -> Left e-    Right (TColumn left') -> case mapColumn (`f` right) left' of-        Left (TypeMismatchException context) ->-            Left $-                TypeMismatchException-                    ( context-                        { callingFunctionName = Just "interpret"-                        , errorColumnName = Just (show expr)-                        }-                    )-        Left e -> Left e-        Right res -> pure $ TColumn res-interpret df expr@(BinaryOp _ (f :: c -> d -> e) left right) = case interpret @c df left of-    Left (TypeMismatchException context) ->-        Left $-            TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show left)-                    }-                )-    Left e -> Left e-    Right (TColumn left') -> case interpret @d df right of-        Left (TypeMismatchException context) ->-            Left $-                TypeMismatchException-                    ( context-                        { callingFunctionName = Just "interpret"-                        , errorColumnName = Just (show right)-                        }-                    )-        Left e -> Left e-        Right (TColumn right') -> case zipWithColumns f left' right' of-            Left (TypeMismatchException context) ->-                Left $-                    TypeMismatchException-                        ( context-                            { callingFunctionName = Just "interpret"-                            , errorColumnName = Just (show expr)-                            }-                        )-            Left e -> Left e-            Right res -> pure $ TColumn res-interpret df expression@(AggVector expr op (f :: v b -> c)) = case interpret @b df expr of-    Left (TypeMismatchException context) ->-        Left $+interpret df (Lit value) = case sUnbox @a of+    -- Specialize the creation of unboxed columns to avoid an extra allocation.+    STrue -> pure $ TColumn $ fromUnboxedVector $ VU.replicate (numRows df) value+    SFalse -> pure $ TColumn $ fromVector $ V.replicate (numRows df) value+interpret df (Col name) = maybe columnNotFound (pure . TColumn) (getColumn name df)+  where+    columnNotFound = Left $ ColumnNotFoundException name "" (M.keys $ columnIndices df)+-- Unary operations.+interpret df expr@(UnaryOp _ (f :: c -> d) value) = first (handleInterpretException (show expr)) $ do+    (TColumn value') <- interpret @c df value+    fmap TColumn (mapColumn f value')+-- Variations of binary operations.+interpret df expr@(BinaryOp _ (f :: c -> d -> e) left right) = first (handleInterpretException (show expr)) $ case (left, right) of+    (Lit left, Lit right) -> interpret df (Lit (f left right))+    (Lit left, right) -> do+        -- If we have a literal then we don't have to materialise+        -- the column.+        (TColumn value') <- interpret @d df right+        fmap TColumn (mapColumn (f left) value')+    (left, Lit right) -> do+        -- Same as the above except the right side is the+        -- literl.+        (TColumn value') <- interpret @c df left+        fmap TColumn (mapColumn (`f` right) value')+    (_, _) -> do+        -- In the general case we interpret and zip.+        (TColumn left') <- interpret @c df left+        (TColumn right') <- interpret @d df right+        fmap TColumn (zipWithColumns f left' right')+-- Conditionals+interpret df expr@(If cond l r) = first (handleInterpretException (show expr)) $ do+    (TColumn conditions) <- interpret @Bool df cond+    (TColumn left) <- interpret @a df l+    (TColumn right) <- interpret @a df r+    let branch (c :: Bool) (l' :: a, r' :: a) = if c then l' else r'+    fmap TColumn (zipWithColumns branch conditions (zipColumns left right))+interpret df expression@(AggVector expr op (f :: v b -> c)) = do+    (TColumn column) <- interpret @b df expr+    -- Helper for errors. Should probably find a way of throwing this+    -- without leaking the fact that we use `Vector` to users.+    let aggTypeError expected =             TypeMismatchException-                ( context-                    { callingFunctionName = Just "interpret"-                    , errorColumnName = Just (show expr)+                ( MkTypeErrorContext+                    { userType = Right (typeRep @(v b))+                    , expectedType = Left expected :: Either String (TypeRep ())+                    , callingFunctionName = Just "interpret"+                    , errorColumnName = Nothing                     }                 )-    Left e -> Left e-    Right (TColumn column) -> case column of-        (BoxedColumn col) -> case testEquality (typeRep @(v b)) (typeOf col) of-            Just Refl -> interpret @c df (Lit (f col))-            Nothing ->-                Left $-                    TypeMismatchException-                        ( MkTypeErrorContext-                            { userType = Right (typeRep @(v b))-                            , expectedType = Right (typeOf col)-                            , callingFunctionName = Just "interpret"-                            , errorColumnName = Nothing-                            }-                        )-        (OptionalColumn col) -> case testEquality (typeRep @(v b)) (typeOf col) of-            Just Refl -> interpret @c df (Lit (f col))-            Nothing ->-                Left $-                    TypeMismatchException-                        ( MkTypeErrorContext-                            { userType = Right (typeRep @(v b))-                            , expectedType = Right (typeOf col)-                            , callingFunctionName = Just "interpret"-                            , errorColumnName = Nothing-                            }-                        )-        (UnboxedColumn (col :: VU.Vector d)) -> case testEquality (typeRep @(v b)) (typeOf col) of+    let processColumn ::+            (Columnable d) => d -> Either DataFrameException (TypedColumn a)+        processColumn col = case testEquality (typeRep @(v b)) (typeOf col) of             Just Refl -> interpret @c df (Lit (f col))-            Nothing -> case testEquality (typeRep @b) (typeRep @d) of-                Just Refl -> interpret @c df (Lit (f (V.convert col)))-                Nothing ->-                    Left $-                        TypeMismatchException-                            ( MkTypeErrorContext-                                { userType = Right (typeRep @(v b))-                                , expectedType = Right (typeOf col)-                                , callingFunctionName = Just "interpret"-                                , errorColumnName = Nothing-                                }-                            )+            Nothing -> Left $ aggTypeError (show (typeOf col))+    case column of+        (BoxedColumn col) -> processColumn col+        (OptionalColumn col) -> processColumn col+        (UnboxedColumn col) -> processColumn col interpret df expression@(AggReduce expr op (f :: a -> a -> a)) = case interpret @a df expr of     Left (TypeMismatchException context) ->         Left $@@ -1271,3 +1142,22 @@ eSize (AggVector expr op _) = eSize expr + 1 eSize (AggReduce expr op _) = eSize expr + 1 eSize (AggFold expr op _ _) = eSize expr + 1++-- Helpers+mkTypeMismatchException ::+    (Typeable a, Typeable b) =>+    Maybe String -> Maybe String -> TypeErrorContext a b -> DataFrameException+mkTypeMismatchException callPoint errorLocation context =+    TypeMismatchException+        ( context+            { callingFunctionName = callPoint+            , errorColumnName = errorLocation+            }+        )++handleInterpretException :: String -> DataFrameException -> DataFrameException+handleInterpretException errorLocation (TypeMismatchException context) = mkTypeMismatchException (Just "interpret") (Just errorLocation) context+handleInterpretException _ e = e++numRows :: DataFrame -> Int+numRows df = fst (dataframeDimensions df)

@@ -0,0 +1,50 @@+{-# LANGUAGE ExplicitNamespaces #-}+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE GADTs #-}+{-# LANGUAGE RankNTypes #-}+{-# LANGUAGE ScopedTypeVariables #-}+{-# LANGUAGE TupleSections #-}++module DataFrame.Monad where++import DataFrame (DataFrame)+import qualified DataFrame as D+import DataFrame.Internal.Column (Columnable)+import DataFrame.Internal.Expression (Expr)++import qualified Data.Text as T++-- A re-implementation of the state monad.+-- `mtl` might be too heavy a dependency just to get+-- a single monad instance.+newtype FrameM a = FrameM {runFrameM_ :: DataFrame -> (DataFrame, a)}++instance Functor FrameM where+    fmap f (FrameM g) = FrameM $ \df ->+        let (df', x) = g df+         in (df', f x)++instance Applicative FrameM where+    pure x = FrameM (,x)+    FrameM ff <*> FrameM fx = FrameM $ \df ->+        let (df1, f) = ff df+            (df2, x) = fx df1+         in (df2, f x)++instance Monad FrameM where+    FrameM g >>= f = FrameM $ \df ->+        let (df1, x) = g df+            FrameM h = f x+         in h df1++deriveM :: (Columnable a) => T.Text -> Expr a -> FrameM (Expr a)+deriveM name expr = FrameM $ \df ->+    let df' = D.derive name expr df+     in (df', expr) -- or (df', F.col @a name)++filterWhereM :: Expr Bool -> FrameM ()+filterWhereM p = FrameM $ \df ->+    (D.filterWhere p df, ())++runFrameM :: DataFrame -> FrameM a -> DataFrame+runFrameM df action = fst (runFrameM_ action df)

@@ -62,13 +62,27 @@         column' <- mapColumn f column         pure $ insertColumn columnName column' d -{- | O(k) Apply a function to a combination of columns in a dataframe and+{- | O(k) Apply a function to an expression in a dataframe and add the result into `alias` column. -} derive :: forall a. (Columnable a) => T.Text -> Expr a -> DataFrame -> DataFrame derive name expr df = case interpret @a df (normalize expr) of     Left e -> throw e     Right (TColumn value) -> insertColumn name value df++{- | O(k) Apply a function to an expression in a dataframe and+add the result into `alias` column but++==== __Examples__++>>> (z, df') = deriveWithExpr "z" (F.col @Int "x" + F.col "y") df+>>> filterWhere (z .>= 50)+-}+deriveWithExpr ::+    forall a. (Columnable a) => T.Text -> Expr a -> DataFrame -> (Expr a, DataFrame)+deriveWithExpr name expr df = case interpret @a df (normalize expr) of+    Left e -> throw e+    Right (TColumn value) -> (Col name, insertColumn name value df)  deriveMany :: [NamedExpr] -> DataFrame -> DataFrame deriveMany exprs df =

@@ -91,7 +91,7 @@                     , signum                     ]                ]-            ++ [ F.pow i p+            ++ [ F.pow p i                | p <- existingPrograms                , i <- [2 .. 6]                ]

@@ -3,11 +3,12 @@  module Functions where +import qualified DataFrame as D import DataFrame.Functions (-    col,     sanitize,  )-import DataFrame.Synthesis (generatePrograms)+import qualified DataFrame.Functions as F+import qualified DataFrame.Internal.Column as DI import Test.HUnit  -- Test cases for the sanitize function@@ -60,20 +61,26 @@                 "_____"                 (sanitize "***")         ]+df :: D.DataFrame+df =+    D.fromNamedColumns+        [("A", DI.fromList [(1 :: Int) .. 10])] -generateProgramsCalledWithNoExistingPrograms :: Test-generateProgramsCalledWithNoExistingPrograms =+testSum :: Test+testSum =     TestCase         ( assertEqual-            "generatePrograms called with no existing programs"-            [col @Double "x"]-            (generatePrograms True [] [col "x"] [] [])+            "Sum first 10 numbers"+            ( D.fromNamedColumns+                [ ("A", DI.fromList [(1 :: Int) .. 10])+                , ("sum", DI.fromList (replicate 10 (55 :: Int)))+                ]+            )+            (D.derive "sum" (F.sum (F.col @Int "A")) df)         )  tests :: [Test] tests =     [ TestLabel "sanitizeIdentifiers" sanitizeIdentifiers-    , TestLabel-        "generateProgramsCalledWithNoExistingPrograms"-        generateProgramsCalledWithNoExistingPrograms+    , TestLabel "testSum" testSum     ]

@@ -534,10 +534,10 @@         ( assertEqual             "mt_cars"             mtCarsDataset-            (unsafePerformIO (D.readParquet "./data/mtcars.parquet"))+            (unsafePerformIO (D.readParquet "./tests/data/mtcars.parquet"))         )  -- Uncomment to run parquet tests. -- Currently commented because they don't run with github CI tests :: [Test]-tests = [] -- [allTypesPlain, allTypesPlainSnappy, allTypesDictionary, mtCars]+tests = [allTypesPlain, allTypesPlainSnappy, allTypesDictionary, mtCars]

binary file changed (absent → 1698 bytes)

binary file changed (absent → 1851 bytes)

binary file changed (absent → 1736 bytes)

binary file changed (absent → 454233 bytes)

binary file changed (absent → 811756 bytes)

binary file changed (absent → 478 bytes)

binary file changed (absent → 3070 bytes)

binary file changed (absent → 324 bytes)

binary file changed (absent → 4104 bytes)

binary file changed (absent → 15659 bytes)

binary file changed (absent → 400 bytes)

binary file changed (absent → 1647 bytes)

binary file changed (absent → 1643 bytes)

binary file changed (absent → 2885 bytes)

binary file changed (absent → 41421 bytes)

binary file changed (absent → 3380 bytes)

binary file changed (absent → 41421 bytes)

binary file changed (absent → 1165 bytes)

binary file changed (absent → 413 bytes)

binary file changed (absent → 72971 bytes)

binary file changed (absent → 68353 bytes)

binary file changed (absent → 11567 bytes)

binary file changed (absent → 13528 bytes)

binary file changed (absent → 3072 bytes)

binary file changed (absent → 635 bytes)

binary file changed (absent → 4335 bytes)

binary file changed (absent → 677 bytes)

binary file changed (absent → 537 bytes)

binary file changed (absent → 501 bytes)

binary file changed (absent → 489 bytes)

binary file changed (absent → 702 bytes)

binary file changed (absent → 358859 bytes)

binary file changed (absent → 595 bytes)

binary file changed (absent → 478 bytes)

binary file changed (absent → 3829 bytes)

binary file changed (absent → 591 bytes)

binary file changed (absent → 495 bytes)

binary file changed (absent → 4325 bytes)

binary file changed (absent → 2526 bytes)

binary file changed (absent → 797 bytes)

binary file changed (absent → 380836 bytes)

binary file changed (absent → 825 bytes)

binary file changed (absent → 4564 bytes)

binary file changed (absent → 329 bytes)

binary file changed (absent → 2850 bytes)

binary file changed (absent → 881 bytes)

binary file changed (absent → 1324 bytes)

binary file changed (absent → 53040 bytes)

binary file changed (absent → 1228 bytes)

binary file changed (absent → 3186 bytes)

binary file changed (absent → 502 bytes)

binary file changed (absent → 3896 bytes)

binary file changed (absent → 461 bytes)

binary file changed (absent → 539 bytes)

binary file changed (absent → 720456 bytes)

binary file changed (absent → 504 bytes)

binary file changed (absent → 816 bytes)

binary file changed (absent → 662 bytes)

binary file changed (absent → 1296 bytes)

binary file changed (absent → 822 bytes)

binary file changed (absent → 814 bytes)

binary file changed (absent → 192 bytes)

binary file changed (absent → 660 bytes)

binary file changed (absent → 1361 bytes)

binary file changed (absent → 1051 bytes)