sparkle 0.2 → 0.3

raw patch · 15 files changed

+92/−34 lines, 15 filesdep +jnidep +jvmdep −inline-java

Dependencies added: jni, jvm

Dependencies removed: inline-java

Files

• README.md +87/−19
• build/libs/sparkle.jar binary
• cbits/bootstrap.c +1/−1
• sparkle.cabal +4/−3
• src/Control/Distributed/Spark/Closure.hs +0/−1
• src/Control/Distributed/Spark/Context.hs +0/−1
• src/Control/Distributed/Spark/ML/Feature/CountVectorizer.hs +0/−1
• src/Control/Distributed/Spark/ML/Feature/RegexTokenizer.hs +0/−1
• src/Control/Distributed/Spark/ML/Feature/StopWordsRemover.hs +0/−1
• src/Control/Distributed/Spark/ML/LDA.hs +0/−1
• src/Control/Distributed/Spark/PairRDD.hs +0/−1
• src/Control/Distributed/Spark/RDD.hs +0/−1
• src/Control/Distributed/Spark/SQL/Context.hs +0/−1
• src/Control/Distributed/Spark/SQL/DataFrame.hs +0/−1
• src/Control/Distributed/Spark/SQL/Row.hs +0/−1

@@ -5,10 +5,12 @@ *Sparkle [spär′kəl]:* a library for writing resilient analytics applications in Haskell that scale to thousands of nodes, using [Spark][spark] and the rest of the Apache ecosystem under the hood.+See [this blog post][hello-sparkle] for the details.  **This is an early tech preview, not production ready.**  [spark]: http://spark.apache.org/+[hello-sparkle]: http://blog.tweag.io/posts/2016-02-25-hello-sparkle.html  ## Getting started @@ -16,14 +18,14 @@  ``` $ stack build hello-$ stack exec sparkle package sparkle-example-hello-$ spark-submit --master 'local[1]' sparkle-example-hello.jar+$ stack exec -- sparkle package sparkle-example-hello+$ stack exec -- spark-submit --master 'local[1]' sparkle-example-hello.jar ```  **Requirements:**-* the [Stack][stack] build tool;+* the [Stack][stack] build tool (version 1.2 or above); * either, the [Nix][nix] package manager,-* or, OpenJDK, Gradle and Spark >= 1.6 installed from your distro.+* or, OpenJDK, Gradle and Spark (version 1.6) installed from your distro.  To run a Spark application the process is as follows: @@ -40,36 +42,102 @@ To build:  ```-$ stack [--nix] build+$ stack build ``` -You can optionally pass `--nix` to all Stack commands to ask Nix to-make Spark and Gradle available in a local sandbox for good build-results reproducibility. Otherwise you'll need these installed through-your OS distribution's package manager for the next steps (and you'll-need to tell Stack how to find the JVM header files and shared-libraries).+You can optionally get Stack to download Spark and Gradle in a local+sandbox (using [Nix][nix]) for good build results reproducibility.+**This is the recommended way to build sparkle.** Alternatively,+you'll need these installed through your OS distribution's package+manager for the next steps (and you'll need to tell Stack how to find+the JVM header files and shared libraries). -To package your app (omit the square bracket part entirely if you're-not using `--nix`):+To use Nix, set the following in your `~/.stack/config.yaml` (or pass+`--nix` to all Stack commands, see the [Stack manual][stack-nix] for+more): +```yaml+nix:+  enable: true ```-$ [stack --nix exec --] sparkle package <app-executable-name>++To package your app as a JAR directly consumable by Spark:+ ```+$ stack exec -- sparkle package <app-executable-name>+```  Finally, to run your application, for example locally:  ```-$ [stack --nix exec --] spark-submit --master 'local[1]' <app-executable-name>.jar+$ stack exec -- spark-submit --master 'local[1]' <app-executable-name>.jar ``` -See [here][spark-submit] for other options, including lauching-a [whole cluster from scratch on EC2][spark-ec2].+The `<app-executable-name>` is any executable name as given in the+`.cabal` file for your app. See apps in the [apps/](apps/) folder for+examples. +See [here][spark-submit] for other options, including launching+a [whole cluster from scratch on EC2][spark-ec2]. This+[blog post][tweag-blog-haskell-paas] shows you how to get started on+the [Databricks hosted platform][databricks] and on+[Amazon's Elastic MapReduce][aws-emr].+ [stack]: https://github.com/commercialhaskell/stack-[spark-submit]: http://spark.apache.org/docs/latest/submitting-applications.html-[spark-ec2]: http://spark.apache.org/docs/latest/ec2-scripts.html+[stack-nix]: https://docs.haskellstack.org/en/stable/nix_integration/#configuration+[spark-submit]: http://spark.apache.org/docs/1.6.2/submitting-applications.html+[spark-ec2]: http://spark.apache.org/docs/1.6.2/ec2-scripts.html [nix]: http://nixos.org/nix+[tweag-blog-haskell-paas]: http://blog.tweag.io/posts/2016-06-20-haskell-compute-paas-with-sparkle.html+[databricks]: https://databricks.com/+[aws-emr]: https://aws.amazon.com/emr/++### Non-Linux OSes++Sparkle is not currently supported on non-linux OSes, e.g. Mac OS X or Windows. If you want to build and use it from a machine using+such an OS, you can use the provided `Dockerfile` and build everything in [docker](http://docker.io):++```+$ docker build -t sparkle .+```++will create an image named `sparkle` containing everything that's+needed to build sparkle and Spark applications: Stack, Java 8, Gradle.++This image can be used to build sparkle then package and run applications:++```+# stack --docker --docker-image sparkle build+...+```++Note that you will need to edit the `stack.yaml` file to point to+include directories and libraries for building the C bits that+interact with the JVM:++```+extra-include-dirs:+  - '/usr/lib/jvm/java-1.8.0-openjdk-amd64/include'+  - '/usr/lib/jvm/java-1.8.0-openjdk-amd64/include/linux'+extra-lib-dirs:+  - '/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre/lib/amd64/server/'+```++Once everything is built you can generate a spark package and run it using `sparkle`'s command-line:++```+# stack --docker --docker-image sparkle exec sparkle package sparkle-example-hello+```++## How it works++sparkle is a tool for creating self-contained Spark applications in+Haskell. Spark applications are typically distributed as JAR files, so+that's what sparkle creates. We embed Haskell native object code as+compiled by GHC in these JAR files, along with any shared library+required by this object code to run. Spark dynamically loads this+object code into its address space at runtime and interacts with it+via the Java Native Interface (JNI).  ## Troubleshooting 

binary file changed (11704 → 11702 bytes)

@@ -65,7 +65,7 @@ 	exitFn = bypass_exit; 	/* Set a control prompt just before calling main. If main() 	 * calls longjmp(), then the exit code of the call to main()-	 * below it must have been zero so just return without further+	 * below must have been zero, so just return without further 	 * ceremony. 	 */ 	if(setjmp(bootstrap_env)) return;

@@ -1,5 +1,5 @@ name:                sparkle-version:             0.2+version:             0.3 synopsis:            Distributed Apache Spark applications in Haskell description:         See README.md license:             BSD3@@ -49,11 +49,12 @@     Control.Distributed.Spark.SQL.Row     Control.Distributed.Spark.RDD   build-depends:-    base >=4.8 && <4.9,+    base >=4.8 && <5,     binary >=0.7,     bytestring >=0.10,     distributed-closure >=0.3,-    inline-java >=0.1,+    jni >=0.1,+    jvm >=0.1,     singletons >= 2.0,     text >=1.2,     vector >=0.11

@@ -4,7 +4,6 @@ {-# LANGUAGE FlexibleInstances #-} {-# LANGUAGE MultiParamTypeClasses #-} {-# LANGUAGE OverloadedStrings #-}-{-# LANGUAGE PolyKinds #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE StaticPointers #-} {-# LANGUAGE TemplateHaskell #-}

@@ -7,7 +7,6 @@ module Control.Distributed.Spark.Context where  import Data.Text (Text, pack, unpack)-import Foreign.JNI import Language.Java  newtype SparkConf = SparkConf (J ('Class "org.apache.spark.SparkConf"))

@@ -12,7 +12,6 @@ import Data.Int import Data.Text (Text) import Foreign.C.Types-import Foreign.JNI import Language.Java  newtype CountVectorizer = CountVectorizer (J ('Class "org.apache.spark.ml.feature.CountVectorizer"))

@@ -8,7 +8,6 @@  import Control.Distributed.Spark.SQL.DataFrame import Data.Text (Text)-import Foreign.JNI import Language.Java  newtype RegexTokenizer = RegexTokenizer (J ('Class "org.apache.spark.ml.feature.RegexTokenizer"))

@@ -8,7 +8,6 @@  import Control.Distributed.Spark.SQL.DataFrame import Data.Text (Text)-import Foreign.JNI import Language.Java  newtype StopWordsRemover = StopWordsRemover (J ('Class "org.apache.spark.ml.feature.StopWordsRemover"))

@@ -10,7 +10,6 @@ import Control.Distributed.Spark.PairRDD import Data.Int import Foreign.C.Types-import Foreign.JNI import Language.Java  newtype LDA = LDA (J ('Class "org.apache.spark.mllib.clustering.LDA"))

@@ -10,7 +10,6 @@ import Control.Distributed.Spark.RDD import Data.Int import Data.Text (Text)-import Foreign.JNI import Language.Java  newtype PairRDD a b = PairRDD (J ('Class "org.apache.spark.api.java.JavaPairRDD"))

@@ -12,7 +12,6 @@ import Control.Distributed.Spark.Context import Data.Int import Data.Text (Text)-import Foreign.JNI import Language.Java  import qualified Data.Text as Text

@@ -7,7 +7,6 @@ module Control.Distributed.Spark.SQL.Context where  import Control.Distributed.Spark.Context-import Foreign.JNI import Language.Java  newtype SQLContext = SQLContext (J ('Class "org.apache.spark.sql.SQLContext"))

@@ -9,7 +9,6 @@ import Control.Distributed.Spark.SQL.Context import Control.Distributed.Spark.SQL.Row import Data.Text (Text)-import Foreign.JNI import Language.Java  newtype DataFrame = DataFrame (J ('Class "org.apache.spark.sql.DataFrame"))

@@ -6,7 +6,6 @@  import Control.Distributed.Spark.PairRDD import Control.Distributed.Spark.RDD-import Foreign.JNI import Language.Java  newtype Row = Row (J ('Class "org.apache.spark.sql.Row"))