LambdaNet 0.1.0.1 → 0.2.0.0

raw patch · 6 files changed

+369/−18 lines, 6 filesPVP ok

version bump matches the API change (PVP)

API changes (from Hackage documentation)

Files

• Changelog +6/−0
• LambdaNet.cabal +10/−3
• Network/Layer.hs +9/−2
• Network/Network.hs +34/−4
• Network/Trainer.hs +73/−9
• README.md +237/−0

@@ -0,0 +1,6 @@+0.2.0.0 - Introduced selection functions and a training function with a+          stop condition.++0.1.0.1 - Updated the Cabal file to fix failing builds++0.1.0.0 - Initial build

@@ -2,10 +2,15 @@ -- documentation, see http://haskell.org/cabal/users-guide/  name:                LambdaNet-version:             0.1.0.1+version:             0.2.0.0 synopsis:            A configurable and extensible neural network library-description:         LambdaNet is an artificial neural network library that allows-                     users to compose their own networks from function primitives.+description: {+LambdaNet is an artificial neural network library that allows+users to compose their own networks from function primitives.+.+Documentation and nightly builds for LambdaNet can be found+at (<http://github.com/jbarrow/LambdaNet>).+} license:             MIT license-file:        LICENSE author:              Brent Baumgartner, Alex Thomas, Harang Ju, Joseph Barrow@@ -14,6 +19,8 @@ category:            Machine Learning build-type:          Simple cabal-version:       >=1.8+extra-source-files:  README.md+                     Changelog  library   exposed-modules:     Network.Network, Network.Neuron, Network.Layer, Network.Trainer

@@ -12,6 +12,7 @@ , showableToLayer  , createLayer+, scaleLayer , connectFully , randomList , boxMuller@@ -73,12 +74,18 @@   Layer (randomMatrix * (connectivity i j))         (randomVector * bias)         (neuronDef layerDef)-  where randomMatrix = (i >< j) (randomList t g)-        randomVector = i |> (randomList t g)+  where randomMatrix = (i >< j) (randomList t g')+        randomVector = i |> (randomList t g'')         i = neuronCount layerDef'         j = neuronCount layerDef         connectivity = connect layerDef'         bias = i |> (repeat 1) -- bias connectivity (full)+        (g', g'') = split g++scaleLayer :: (Floating (Vector a), Container Vector a)+  => a -> Layer a -> Layer a+scaleLayer factor l =+  Layer (factor `scale` (weightMatrix l)) (factor `scale` (biasVector l)) (neuron l)  -- | The connectFully function takes the number of input neurons for a layer, i, --   and the number of output neurons of a layer, j, and returns an i x j

@@ -1,11 +1,14 @@-{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE FlexibleContexts,+             UndecidableInstances #-}  module Network.Network ( Network(..)-, TrainingData  , createNetwork , loadNetwork+, emptyNetwork+, isEmptyNetwork+, addNetworks , predict , apply , saveNetwork@@ -17,12 +20,20 @@ import Numeric.LinearAlgebra import qualified Data.ByteString.Lazy as B import System.IO+import Data.Monoid (Monoid(..)) import Data.Binary (encode, decode, Binary(..))  -- | Networks are constructed front to back. Start by adding an input layer, --   then each hidden layer, and finally an output layer. data Network a = Network { layers :: [Layer a] } +-- | We gain the ability to combine two networks of the same proportions+--   by abstracting a network as a monoid. This is useful in backpropagation+--   for batch training+instance (Product a, Container Vector a, Floating (Vector a)) => Monoid (Network a) where+  mempty = emptyNetwork+  mappend = addNetworks+ -- | A tuple of (input, expected output) type TrainingData a = (Vector a, Vector a) @@ -35,8 +46,27 @@ createNetwork t g (layerDef : []) = Network [] createNetwork t g (layerDef : layerDef' : otherLayerDefs) =   Network (layer : layers restOfNetwork)-  where layer = createLayer t g layerDef layerDef'-        restOfNetwork = createNetwork t g (layerDef' : otherLayerDefs)+  where layer = createLayer t g' layerDef layerDef'+        restOfNetwork = createNetwork t g'' (layerDef' : otherLayerDefs)+        (g', g'') = split g++-- | Our Unit, an empty network with no layers+emptyNetwork :: Network a+emptyNetwork = Network []++-- | A boolean to check if the network is the unit network or not+isEmptyNetwork :: Network a -> Bool+isEmptyNetwork n = length (layers n) == 0++-- | A function to combine two networks+addNetworks :: (Floating (Vector a), Container Vector a, Product a)+  => Network a -> Network a -> Network a+addNetworks n1 n2 = if isEmptyNetwork n1 then n2 else+  if isEmptyNetwork n2 then n1 else+    Network $ zipWith combineLayers (layers n1) (layers n2)+  where combineLayers l1 l2 =+          Layer ((weightMatrix l1) + (weightMatrix l2))+          ((biasVector l1) + (biasVector l2)) (neuron l1)  -- | Predict folds over each layer of the network using the input vector as the --   first value of the accumulator. It operates on whatever network you pass in.

@@ -4,8 +4,14 @@ ( BackpropTrainer(..) , CostFunction , CostFunction'+, TrainingData , Selection+, TrainCompletionPredicate +, trainNTimes+, trainUntilErrorLessThan+, trainUntil+ , quadraticCost , quadraticCost' , minibatch@@ -14,6 +20,8 @@ , inputs , outputs , deltas+, hiddenDeltas+, calculateNablas , fit , evaluate ) where@@ -46,9 +54,57 @@ -- | A CostFunction' (derivative) is used in backPropagation type CostFunction' a = Vector a -> Vector a -> Vector a +-- | A tuple of (input, expected output)+type TrainingData a = (Vector a, Vector a)+ -- | A selection function for performing gradient descent type Selection a = [TrainingData a] -> [[TrainingData a]] +-- | A predicate (given a network, trainer, a list of training+--   data, and the number of [fit]s performed) that+--   tells the trainer to stop training+type TrainCompletionPredicate a = Network a -> BackpropTrainer a -> [TrainingData a] -> Int -> Bool++-- | Given a network, a trainer, a list of training data,+--   and N, this function trains the network with the list of+--   training data N times+trainNTimes :: (Floating (Vector a), Container Vector a, Product a)+  => Network a  -> BackpropTrainer a -> Selection a -> [TrainingData a] -> Int -> Network a+trainNTimes network trainer s dat n =+  trainUntil network trainer s dat completion 0+  where completion _ _ _ n' = (n == n')++-- | Given a network, a trainer, a list of training data,+--   and an error value, this function trains the network with the list of+--   training data until the error of the network (calculated+--   by averaging the errors of each training data) is less than+--   the given error value+trainUntilErrorLessThan :: (Floating (Vector a), Container Vector a, Product a, Ord a)+  => Network a  -> BackpropTrainer a -> Selection a -> [TrainingData a] -> a -> Network a+trainUntilErrorLessThan network trainer s dat err =+  trainUntil network trainer s dat (networkErrorLessThan err) 0++-- | This function returns true if the error of the network is less than+--   a given error value, given a network, a trainer, a list of+--   training data, and a counter (should start with 0)+--   Note: Is there a way to have a counter with a recursive function+--         without providing 0?+networkErrorLessThan :: (Floating (Vector a), Container Vector a, Product a, Ord a)+  => a -> Network a -> BackpropTrainer a -> [TrainingData a] -> Int -> Bool+networkErrorLessThan err network trainer dat _ = meanError < err+  where meanError = (sum errors) / fromIntegral (length errors)+        errors = map (evaluate trainer network) dat++-- | This function trains a network until a given TrainCompletionPredicate+--   is satisfied.+trainUntil :: (Floating (Vector a), Container Vector a, Product a)+  => Network a -> BackpropTrainer a -> Selection a -> [TrainingData a] -> TrainCompletionPredicate a -> Int -> Network a+trainUntil network trainer s dat completion n =+  if completion network trainer dat n+    then network+    else trainUntil network' trainer s dat completion (n+1)+      where network' = fit s trainer network dat+ -- | The quadratic cost function (1/2) * sum (y - a) ^ 2 quadraticCost :: (Floating (Vector a), Container Vector a)   => Vector a -> Vector a -> a@@ -80,23 +136,31 @@ -- | Perform backpropagation on a single training data instance. backprop :: (Floating (Vector a), Container Vector a, Product a)   => BackpropTrainer a -> Network a -> [TrainingData a] -> Network a-backprop t n (e:es) = updateNetwork t n-  (deltas t n e) (outputs (fst e) n)+backprop t n es =+  updateNetwork (length es) t (foldl (calculateNablas t n) emptyNetwork es) n --- | Update the weights and biases of a network given a list of deltas+-- | Given the size of the minibatch, the trainer, the nablas for each layer, given+--   as a network, and the network itself, return a network with updated wieghts. updateNetwork :: (Floating (Vector a), Container Vector a, Product a)-  => BackpropTrainer a -> Network a -> [Vector a] -> [Vector a] -> Network a-updateNetwork t n deltas os =-  Network $ map (updateLayer t) (zip3 (layers n) deltas os)+  => Int -> BackpropTrainer a -> Network a -> Network a -> Network a+updateNetwork mag t nablas n = addNetworks n+  (Network $ map (scaleLayer $ -1 * (eta t) / (fromIntegral mag)) (layers nablas)) +-- | Calculate the nablas for a minibatch and return them as a network (so each+--   weight and bias gets its own nabla).+calculateNablas :: (Floating (Vector a), Container Vector a, Product a)+  => BackpropTrainer a -> Network a -> Network a -> TrainingData a -> Network a+calculateNablas t n nablas e = Network $ map (updateLayer t) (zip3 (layers n) ds os)+  where ds = deltas t n e+        os = outputs (fst e) n+ -- | The mapped function to update the weight and biases in a single layer updateLayer :: (Floating (Vector a), Container Vector a, Product a)   => BackpropTrainer a -> (Layer a, Vector a, Vector a) -> Layer a updateLayer t (l, delta, output) = Layer newWeight newBias n   where n = neuron l-        newWeight = (weightMatrix l) --          (eta t) `scale` ((reshape 1 delta) <> (reshape (dim output) output))-        newBias = (biasVector l) - (eta t) `scale` delta+        newWeight = ((reshape 1 delta) <> (reshape (dim output) output))+        newBias = delta  -- | The outputs function scans over each layer of the network and stores the --   activated results

@@ -0,0 +1,237 @@+LambdaNet+=====++LambdaNet is an artificial neural network library written in Haskell+that abstracts network creation, training, and use as higher order+functions. The benefit of this approach is that it provides a framework+in which users can:+  - quickly iterate through network designs by using different functional components+  - experiment by writing small functional components to extend the library++The library comes with a pre-defined set of functions that can be composed+in many ways to operate on real-world data. These will be enumerated later+in the documentation.++## Installation++LambdaNet can be installed through Cabal:++```+cabal update+cabal install LambdaNet+```++## Using LambdaNet++Using LambdaNet to rapidly prototype networks using built-in functions+requires only a minimal level of Haskell knowledge (although getting+the data into the right form may be more difficult). However, extending+the library may require a more in-depth knowledge of Haskell and+functional programming techniques.++You can find a quick example of using the network in `XOR.hs`. Once LambdaNet+is installed, download XOR.hs, and then you can run the file in your REPL to+see the results:++```+runhaskell examples/XOR.hs+```++The rest of this section dissects the XOR network in order to talk about+the design of LambdaNet.++### Training Data++Before you can train or use a network, you must have training data. The+training data is a tuple of vectors, the first value being the input+to the network, and the second value being the expected output.++For the XOR network, the data is easily hardcoded:++```+let trainData = [+  (fromList [0.0, 0.0], fromList [0.0]),+  (fromList [0.0, 1.0], fromList [1.0]),+  (fromList [1.0, 0.0], fromList [1.0]),+  (fromList [1.0, 1.0], fromList [0.0])+]+```++However, for any non-trivial application the most difficult work will be+getting the data in this form. Unfortunately, LambdaNet does not currently+have tools to support data handling.++### Layer Definitions++The first step in creating a network is to define a list of layer+definitions. The type layer definition takes a neuron type, a count of+neurons in the layer, and a connectivity function.++Creating the layer definitions for a three-layer XOR network, with+2 neurons in the input layer, 2 hidden neurons, and 1 output neuron+can be done as:++```+let l = LayerDefinition sigmoidNeuron 2 connectFully+let l' = LayerDefinition sigmoidNeuron 2 connectFully+let l'' = LayerDefinition sigmoidNeuron 1 connectFully+```++#### Neuron Types++A neuron is simply defined as an activation function and its derivative,+and the LambdaNet library provides three built-in neuron types:+  - `sigmoidNeuron` - A neuron with a sigmoid activation function+  - `tanhNeuron` - A neuron with a hyperbolic tangent activation function+  - `recluNeuron` - A neuron with a rectified linear activation function++By passing one of these functions into a LayerDefinition, you can+create a layer with neurons of that type.++#### Connectivity++A connectivity function is a bit more opaque. Currently, the library+only provides `connectFully`, a function which creates a fully+connected feed-forward network.++Simply, the connectivity function takes in the number of neurons in layer l+and the number of neurons in layer l + 1, and returns a boolean matrix+of integers (0/1) that represents the connectivity graph of the layers+-- a 0 means two neurons are not connected and a 1 means they are. The+starting weights are defined later.++### Creating the Network++The `createNetwork` function takes in a random transform, an entropy+generator, and a list of layer definitions, and returns a network.++For the XOR network, the createNetwork function is:++```+let n = createNetwork normals (mkStdGen 4) [l, l', l'']+```++Our source of entropy is the very random: `mkStdGen 4`, which will+always result in the same generator.++#### Random Transforms++The random transform function is a transform that operates on a+stream of uniformly distributed random numbers and returns a stream+of floating point numbers.++Currently, the two defined distributions are:+  - `uniforms` - A trivial function that returns a stream of uniformly distributed random numbers+  - `normals` - A slightly less-trivial function that uses the Box-Muller transform to create a stream of numbers ~ N(0, 1)++Work is being done to offer a student t-distribution, which would require+support for a chi-squared distribution transformation.++### Training the Network++In order to train a network, you must create a new trainer:++```+let t = BackpropTrainer (3 :: Float) quadraticCost quadraticCost'+```++The BackpropTrainer type takes in a learning rate, a cost function, and+its derivative.++The actual training of the network, the `fit` function uses the trainer, a+network, and the training data, and returns a new, trained network.+For the XOR network, this is:++```+let n' = trainUntilErrorLessThan n t online dat 0.01+```++LambdaNet provides three training methods:+  - `trainUntil`+  - `trainUntilErrorLessThan`+  - `trainNTimes`++The `trainUntil` function takes a TrainCompletionPredicate (check Network/Trainer.hs)+for more information, and the last two are simply wrappers for the first one that+provide specific predicates.++The calculated error is what is returned by the cost function.++#### Cost Functions++Currently, the only provided cost function is the quadratic error cost function,+`quadraticCost` and its derivative, `quadraticCost'`. I am about to add the+cross-entropy cost function.++#### Selection Functions++Selection functions break up a dataset for each round of training. The currently provided+selection functions are:+  - `minibatch n` - You must provide an n and partially apply it to minibatch to get a valid selection function. This function updates the network after every n passes.+  - `online` - Using this function means that the network updates after every training example.++For small data sets, it's better to use online, while for larger data sets, the training+can occur much faster if you use a reasonably sized minibatch.++### Using the Network++Once the network is trained, you can use it with your test data or+production data:++```+predict (fromList [1, 0]) n'+```++LambdaNet at least attempts to follow a Scikit-Learn style naming scheme+with `fit` and `predict` functions.++### Storing and Loading++Once a network has been trained, the weights and biases can be stored in+a file:++```+saveNetwork "xor.ann" n'+```++By calling `saveNetwork` with a file path, you can save the state of the+network.++Loading a network requires passing in a list of layer definitions+for the original network, but will load all the weights and biases of the+saved network:++```+n'' <- loadNetwork "xor.ann" [l, l', l'']+```++Note that the loadNetwork function returns an IO (Network), you can't simply+call predict or train on the object returned by loadNetwork. Using the+approach in XOR.hs should allow you to work with the returned object.++## Currently Under Development++What has been outlined above is only the first stages of LambdaNet. I intend+to support some additional features, such as:+  - Regularization functions+  - Additional trainer types (RProp, RMSProp)+  - Additional cost functions++### Regularization Functions and Momentum++Standard backprop training is subject to overfitting and falling into local+minima. By providing support for regularization and momentum, LambdaNet+will be able to provide more extensible and robust training.++## Generating the Documentation Images++All the documentation for the network was generated in the following manner. In the docs folder, run:++```+runhaskell docs.hs+python analysis.py+```++Note that I am currently working on removing the Python image analysis+from the library, and switching it with Haskell and gnuplot. I'm also+working on using the generated images in network documentation.