diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,30 @@ Changelog ========= +Version 0.2.3.0 +--------------- + +*May 25, 2018* + + + +* Argument order in `backpropWith` family of functions switched around to + allow for final gradient to be given after-the-fact. **Braking change** + for anyone using any `backpropWith` function. +* As a consequence of the previous change, `backprop` family of functions in + *Explicit* interfaces also all changed argument order. **Breaking change** + only for those using the *Explicit* interfaces. +* Explicit `collectVar` no longer needs a `ZeroFunc` for the container, and + so all versions of `collectVar` and functions that use it (`fmap`, + `liftA2`, `liftA3`, `traverse`, `mapAccumL`, `mapAccumR`) no longer require + `Backprop` or `Num` instances for the final returned container type. This + enables a lot more flexibility in container types. **Breaking change** + only for those using the *Explicit* interfaces. +* `BV` pattern synonym added to *Numeric.Backprop*, abstracting over + application of `splitBV` and `joinBV`. +* `foldr` and `foldl'` added to Prelude modules, for convenience. +* `round` and `fromIntegral'` ("unround") added to Prelude modules. + Version 0.2.2.0 --------------- diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,37 +1,54 @@ -backprop -======== - -[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +[backprop][docs] +================ -[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=2592000)](https://hackage.haskell.org/package/backprop) +[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=86400)](https://hackage.haskell.org/package/backprop) +[![backprop on Stackage LTS 11](http://stackage.org/package/backprop/badge/lts-11)](http://stackage.org/lts-11/package/backprop) +[![backprop on Stackage Nightly](http://stackage.org/package/backprop/badge/nightly)](http://stackage.org/nightly/package/backprop) [![Build Status](https://travis-ci.org/mstksg/backprop.svg?branch=master)](https://travis-ci.org/mstksg/backprop) -[**Introductory blog post**][blog] +[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) +[![Beerpay](https://beerpay.io/mstksg/backprop/badge.svg?style=beer-square)](https://beerpay.io/mstksg/backprop) -[blog]: https://blog.jle.im/entry/introducing-the-backprop-library.html +[**Documentation and Walkthrough**][docs] +[docs]: https://backprop.jle.im + Automatic *heterogeneous* back-propagation. Write your functions to compute your result, and the library will automatically generate functions to compute your gradient. Differs from [ad][] by offering full heterogeneity -- each intermediate step -and the resulting value can have different types. Mostly intended for usage -with gradient descent and other numeric optimization techniques. +and the resulting value can have different types (matrices, vectors, scalars, +lists, etc.). [ad]: http://hackage.haskell.org/package/ad -Currently up on [hackage][] (with 100% documentation coverage), but more -up-to-date documentation is currently rendered [on github pages][docs]! +Useful for applications in [differential programming][dp] and deep learning for +creating and training numerical models, especially as described in this blog +post on [a purely functional typed approach to trainable models][models]. +Overall, intended for the implementation of gradient descent and other numeric +optimization techniques. Comparable to the python library [autograd][]. +[dp]: https://www.facebook.com/yann.lecun/posts/10155003011462143 +[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html +[autograd]: https://github.com/HIPS/autograd + +Currently up on [hackage][], with haddock documentation! However, a proper +library introduction and usage tutorial [is available here][docs]. See also my +[introductory blog post][blog]. You can also find help or support on the +[gitter channel][gitter]. + [hackage]: http://hackage.haskell.org/package/backprop -[docs]: https://mstksg.github.io/backprop +[blog]: https://blog.jle.im/entry/introducing-the-backprop-library.html +[gitter]: https://gitter.im/haskell-backprop/Lobby If you want to provide *backprop* for users of your library, see this **[guide to equipping your library with backprop][library]**. -[library]: https://github.com/mstksg/backprop/wiki/Equipping-your-Library-with-Backprop +[library]: https://backprop.jle.im/06-equipping-your-library.html + MNIST Digit Classifier Example ------------------------------ @@ -61,7 +78,8 @@ Brief example ------------- -(This is a really brief version of my [blog post][blog]) +(This is a really brief version of [the documentation walkthrough][docs] and my +[blog post][blog]) The quick example below describes the running of a neural network with one hidden layer to calculate its squared error with respect to target `targ`, @@ -74,138 +92,88 @@ [lens]: http://hackage.haskell.org/package/lens ```haskell -data Network i h o = Net { _weight1 :: L h i - , _bias1 :: R h - , _weight2 :: L o h - , _bias2 :: R o - } +import Numeric.LinearAlgebra.Static.Backprop +data Network = Net { _weight1 :: L 20 100 + , _bias1 :: R 20 + , _weight2 :: L 5 20 + , _bias2 :: R 5 + } + makeLenses ''Network ``` -Normally, we might write code to "run" a neural network on an input like this: +(`R n` is an n-length vector, `L m n` is an m-by-n matrix, etc., `#>` is +matrix-vector multiplication) +"Running" a network on an input vector might look like this: + ```haskell -neuralNet - :: R i - -> Network i h o - -> R h -neuralNet x n = z +runNet net x = z where - y = logistic $ (n ^. weight1) #> x + (n ^. bias1) - z = logistic $ (n ^. weight2) #> y + (n ^. bias2) + y = logistic $ (net ^^. weight1) #> x + (net ^^. bias1) + z = logistic $ (net ^^. weight2) #> y + (net ^^. bias2) logistic :: Floating a => a -> a logistic x = 1 / (1 + exp (-x)) ``` -(`R i` is an i-length vector, `L h i` is an h-by-i matrix, etc., `#>` is -matrix-vector multiplication, and `^.` is access to a field via lens.) - -When given an input vector and the network, we compute the result of the neural -network ran on the input vector. +And that's it! `neuralNet` is now backpropagatable! -We can write it, instead, using *backprop*: +We can "run" it using `evalBP`: ```haskell -neuralNet - :: Reifies s W - => BVar s (R i) - -> BVar s (Network i h o) - -> BVar s (R o) -neuralNet x n = z - where - y = logistic $ (n ^^. weight1) #> x + (n ^^. bias1) - z = logistic $ (n ^^. weight2) #> y + (n ^^. bias2) - -logistic :: Floating a => a -> a -logistic x = 1 / (1 + exp (-x)) +evalBP2 runNet :: Network -> R 100 -> R 5 ``` -(`#>!` is a backprop-aware version of `#>`, and `^^.` is access to a field via -lens in a `BVar`) +If we write a function to compute errors: -And that's it! `neuralNet` is now backpropagatable! +```haskell +squaredError target output = error `dot` error + where + error = target - output +``` -We can "run" it using `evalBP`: +we can "test" our networks: ```haskell -evalBP (neuralNet (constVar x)) :: Network i h o -> R o +netError target input net = squaredError (auto target) + (runNet net (auto input)) ``` -And we can find the gradient using `gradBP`: +This can be run, again: ```haskell -gradBP (neuralNet (constVar x)) :: Network i h o -> Network i h o +evalBP (netError myTarget myVector) :: Network -> Network ``` -If we write a function to compute errors: +Now, we just wrote a *normal function to compute the error of our network*. +With the *backprop* library, we now also have a way to *compute the gradient*, +as well! ```haskell -netError - :: Reifies s W - => BVar s (R i) - -> BVar s (R o) - -> BVar s (Network i h o) - -> BVar s Double -netError x targ n = norm_2 (neuralNet x - t) +gradBP (netError myTarget myVector) :: Network -> Network ``` -(`norm_2` is a backprop-aware euclidean norm) - Now, we can perform gradient descent! ```haskell gradDescent - :: R i - -> R o - -> Network i h o - -> Network i h o + :: R 100 + -> R 5 + -> Network + -> Network gradDescent x targ n0 = n0 - 0.1 * gradient where - gradient = gradBP (netError (constVar x) (constVar targ)) n0 + gradient = gradBP (netError targ x) n0 ``` Ta dah! We were able to compute the gradient of our error function, just by only saying how to compute *the error itself*. -For a more fleshed out example, see my [blog post][blog] and the [MNIST -tutorial][mnist-lhs] (also [rendered as a pdf][mnist-pdf]) - -Lens Access ------------ - -A lot of the friction of dealing with `BVar s a`s instead of `a`s directly is -alleviated with the lens interface. - -With a lens, you can "view" and "set" items inside a `BVar`, as if they were -the actual values: - -```haskell -(^.) :: a -> Lens' a b -> b -(^^.) :: BVar s a -> Lens' a b -> BVar s b - -(.~) :: Lens' a b -> b -> a -> a -(.~~) :: Lens' a b -> BVar s b -> BVar s a -> BVar s a -``` - -And you can also extract multiple potential targets, as well, using -`Traversal`s and `Prism`s: - -```haskell --- | Actually takes a Traversal, to be more general. --- Can be used to implement "pattern matching" on BVars -(^?) :: a -> Prism' a b -> Maybe ( b) -(^^?) :: BVar s a -> Prism' a b -> Maybe (BVar s b) - -(^..) :: a -> Traversal' a b -> [ b] -(^^..) :: BVar s a -> Traversal' a b -> [BVar s b] -``` - -Note that the library itself has no *lens* dependency, using *[microlens][]* -instead. - -[microlens]: http://hackage.haskell.org/package/microlens +For a more fleshed out example, see [the documentaiton][docs], my [blog +post][blog] and the [MNIST tutorial][mnist-lhs] (also [rendered as a +pdf][mnist-pdf]) Benchmarks ---------- @@ -263,3 +231,7 @@ b. How to support "monadic" operations that depend on results of previous operations? (`ApBP` already exists for situations that don't) + + c. What needs to be done to allow us to automatically do second, + third-order differentiation, as well? This might be useful for certain + ODE solvers which rely on second order gradients and hessians. diff --git a/backprop.cabal b/backprop.cabal --- a/backprop.cabal +++ b/backprop.cabal @@ -2,20 +2,20 @@ -- -- see: https://github.com/sol/hpack -- --- hash: d347cf6994856b821bb3cf3172a4b5ec8f0d39b680e29e39a019d89cf022b2a5 +-- hash: 983ab7f63f7a2d0309dc97c50875836220f6cbd8535600ad8cbbef9fe0672195 name: backprop -version: 0.2.2.0 -synopsis: Heterogeneous automatic differentation (backpropagation) +version: 0.2.3.0 +synopsis: Heterogeneous automatic differentation description: Write your functions to compute your result, and the library will automatically generate functions to compute your gradient. . Implements heterogeneous reverse-mode automatic differentiation, commonly known as "backpropagation". . - See + See for official introduction and documentation. category: Math -homepage: https://github.com/mstksg/backprop#readme +homepage: https://backprop.jle.im bug-reports: https://github.com/mstksg/backprop/issues author: Justin Le maintainer: justin@jle.im @@ -28,6 +28,13 @@ extra-source-files: Build.hs CHANGELOG.md + doc/01-getting-started.md + doc/02-a-detailed-look.md + doc/03-manipulating-bvars.md + doc/04-the-backprop-typeclass.md + doc/05-applications.md + doc/06-equipping-your-library.md + doc/index.md README.md renders/backprop-mnist.md renders/backprop-mnist.pdf diff --git a/doc/01-getting-started.md b/doc/01-getting-started.md new file mode 100644 --- /dev/null +++ b/doc/01-getting-started.md @@ -0,0 +1,227 @@ +--- +title: Getting Started +--- + +Getting Started +=============== + +```haskell top hide +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DeriveGeneric #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE TemplateHaskell #-} +{-# LANGUAGE ViewPatterns #-} + + +import GHC.Generics (Generic) +import GHC.TypeNats +import Inliterate.Import +import Lens.Micro +import Lens.Micro.TH +import Numeric.Backprop.Class +import Numeric.LinearAlgebra.Static (L, R) +import System.Random +import qualified Numeric.LinearAlgebra.Static as H +``` + +*backprop* is a Haskell library available on hackage, so can be used in your +package however way you like to require libraries. Be sure to add it to your +cabal file's (or package.yaml's) build-depends field. + +Automatic Backpropagated Functions +---------------------------------- + +With *backprop*, you can write your functions in Haskell as normal functions: + +```haskell top +import Numeric.Backprop + +myFunc x = sqrt (x * 4) +``` + +They can be run with `evalBP`: + +```haskell eval +evalBP myFunc (9 :: Double) +``` + +And...the twist? You can also get the gradient of your functions! + +```haskell eval +gradBP myFunc (9 :: Double) +``` + +And that's the gist of the entire library: write your functions to compute your +things, and `gradBP` will give you the gradients and derivatives of those +functions. + +### Multiple Same-Type Inputs + +Multiple inputs of the same type can be handled with `sequenceVar`: + +```haskell top +funcOnList (sequenceVar->[x,y,z]) = sqrt (x / y) * z +``` + +```haskell eval +evalBP funcOnList [3,5,-2] :: Double +``` + +```haskell eval +gradBP funcOnList [3,5,-2] :: [Double] +``` + +Heterogeneous Backprop +---------------------- + +But the real magic happens when you mix and match types. Let's make a simple +type representing a feed-forward fully connected artificial neural network with +100 inputs, a single hidden layer of 20 nodes, and 5 outputs: + +```haskell top +data Net = N { _nWeights1 :: L 20 100 + , _nBias1 :: R 20 + , _nWeights2 :: L 5 20 + , _nBias2 :: R 5 + } + deriving (Show, Generic) + +instance Backprop Net + +makeLenses ''Net +``` + +using the `L m n` type from the *[hmatrix][]* library to represent an m-by-n +matrix, and the `R n` type to represent an n-vector. + +[hmatrix]: http://hackage.haskell.org/package/hmatrix + +We can write a function to "run" the network on a `R 100` and get an `R 5` +back, using `^^.` for lens access and `#>` from the *[hmatrix-backprop][]* library for +matrix-vector multiplication: + +[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop + +```haskell top hide +instance Backprop (R n) where + zero = zeroNum + add = addNum + one = oneNum + +instance (KnownNat n, KnownNat m) => Backprop (L n m) where + zero = zeroNum + add = addNum + one = oneNum + +(#>) + :: (KnownNat n, KnownNat m, Reifies s W) + => BVar s (L n m) -> BVar s (R m) -> BVar s (R n) +(#>) = liftOp2 . op2 $ \xs y -> + ( xs H.#> y + , \d -> (d `H.outer` y, H.tr xs H.#> d) + ) + +dot :: (KnownNat n, Reifies s W) => BVar s (R n) -> BVar s (R n) -> BVar s Double +dot = liftOp2 . op2 $ \x y -> + ( x `H.dot` y + , \d -> let d' = H.konst d + in (d' * y, x * d') + ) +``` + +```haskell top +runNet net x = z + where + -- run first layer + y = logistic $ (net ^^. nWeights1) #> x + (net ^^. nBias1) + -- run second layer + z = logistic $ (net ^^. nWeights2) #> y + (net ^^. nBias2) + +logistic :: Floating a => a -> a +logistic x = 1 / (1 + exp (-x)) +``` + +We can *run* this with a network and input vector: + +```haskell top hide +myVector :: R 100 +myVector = H.randomVector 93752345 H.Uniform - 0.5 + +myTarget :: R 5 +myTarget = H.randomVector 93752345 H.Uniform - 0.5 + +myNet :: Net +myNet = N (H.uniformSample 2394834 (-0.5) 0.5) + (H.randomVector 84783451 H.Uniform - 0.5) + (H.uniformSample 9293092 (-0.5) 0.5) + (H.randomVector 64814524 H.Uniform - 0.5) + +instance KnownNat n => AskInliterate (R n) where + askInliterate = answerWith (show . H.extract) +instance AskInliterate Net where + askInliterate = answerWith (unlines . (++ ["-- ..."]) . take 5 . lines . show) +``` + +```haskell eval +evalBP2 runNet myNet myVector +``` + +But --- and here's the fun part --- if we write a "loss function" to evaluate +"how badly" our network has done, using `dot` from the *hmatrix-backprop* +library: + +```haskell top +squaredError target output = error `dot` error + where + error = target - output +``` + +we can "test" our networks: + +```haskell top +netError target input net = squaredError (auto target) + (runNet net (auto input)) +``` + +(more on `auto` later) + +```haskell eval +evalBP (netError myTarget myVector) myNet +``` + +At this point, we've *written a normal function to compute the error of our +network*. And, with the backprop library...we now have a way to compute the +*gradient* of our network's error with respect to all of our weights! + +```haskell eval +gradBP (netError myTarget myVector) myNet +``` + +We can now use the gradient to "[train][]" our network to give the correct +responses given a certain input! This can be done by computing the gradient +for every expected input-output pair, and adjusting the network in the opposite +direction of the gradient every time. + +[train]: https://blog.jle.im/entry/purely-functional-typed-models-1.html + +Main Idea +--------- + +The main pattern of usage for this library is: + +1. Write your function normally to compute something (like the loss function) +2. Use `gradBP` to automatically get the gradient of that something with + respect to your inputs! + +In the case of optimizing models, you: + +1. Write your function normally to compute the thing you want to minimize +2. Use `gradBP` to automatically get the gradient of the thing you want to + minimize with respect to your inputs. Then, adjust your inputs according + to this gradient until you get the perfect minimal result! + +Now that you've had a taste, let's **[look at the details][details]**. You can +also just go ahead and **[jump into the haddock documentation][haddock]**! + +[details]: https://backprop.jle.im/02-a-detailed-look.html +[haddock]: https://hackage.haskell.org/package/backprop diff --git a/doc/02-a-detailed-look.md b/doc/02-a-detailed-look.md new file mode 100644 --- /dev/null +++ b/doc/02-a-detailed-look.md @@ -0,0 +1,127 @@ +--- +title: A Detailed Look +--- + +A Detailed Look +=============== + +```haskell top hide +{-# LANGUAGE FlexibleContexts #-} + +import Numeric.Backprop +``` + +So, what's really going on? + +The BVar +-------- + +The entire library revolves around the `BVar`, a variable holding a +"backpropagatable value". As you use a `BVar`, the *backprop* library will +track how it is used and where you use it. You can use `evalBP` to simply get +the result, but using `gradBP` will perform backpropagation ("reverse-mode +[automatic differentiation][autodiff]") + +[autodiff]: https://en.wikipedia.org/wiki/Automatic_differentiation + +For example, we looked earlier at a function that computes the square root of a +quadrupled number: + +```haskell top +myFunc :: Double + -> Double +myFunc x = sqrt (x * 4) +``` + +As we are using it, its type is "really": + +```haskell top +myFunc' :: Reifies s W + => BVar s Double + -> BVar s Double +myFunc' x = sqrt (x * 4) +``` + +`myFunc'` takes a `BVar s Double` (a `BVar` containing a `Double`) and returns +a new one that is the square root of the quadrupled number. You can think of +the `Reifies s W` as being a necessary constraint that allows backpropagation +to happen. + +`BVar`s have `Num`, `Fractional`, and `Floating` instances, and so can be used +with addition, multiplication, square rooting, etc. The "most general" type of +`myFunc` is `myFunc :: Floating a => a -> a`, and since `BVar s Double` has a +`Floating` instance, you could even just use it directly as a backpropagatable +function. + +This means you can basically treat a `BVar s Double` almost exactly like it was +a `Double` --- you'll practically never tell the difference! `BVar`s also have +`Ord` and `Eq` instances, so you can compare them and branch on the results, +too. + +```haskell top +myAbs :: Reifies s W + => BVar s Double + -> BVar s Double +myAbs x | x < 0 = negate x + | otherwise = x +``` + +The goal of the `BVar` interface is that you should be able to treat a `BVar s +a` (a `BVar` containing an `a`) as if it was an `a`, with no easily noticeable +differences. + +Runners +------- + +The entire point of the library is to write your computation as a normal +function taking a `BVar` (or many) and returning a single `BVar`. Just treat +`BVar`s as if they actually were the value they are containing, and you can't +go wrong. + +Once you do this, you can use `evalBP` to "run" the function itself: + +```haskell +evalBP :: (forall s. Reifies s W => BVar s a -> BVar s b) + -> (a -> b) +``` + +This can be read as taking a `BVar s a -> BVar s b` and returning the `a -> b` +that that function encodes. The RankN type there (the `forall s.`) is mostly +there to prevent leakage of `BVar`s (same as it is used in *Control.Monad.ST* +and `runST`). It ensures that no `BVar`s "escape" the function somehow. + +`evalBP` is extremely efficient, and usually carries virtually zero overhead +over writing your function directly on your values without `BVar`s. + +*But*, the more interesting thing of course is computing the *gradient* of your +function. This is done with `gradBP`: + +```haskell +gradBP :: (Backprop a, Backprop b) + => (forall s. Reifies s W => BVar s a -> BVar s b) + -> a + -> a +``` + +Which takes a `BVar s a -> BVar s b` backpropagatable function and an input, +and returns *the gradient at that input*. It gives the direction of greatest +positive change (in the output) of your input, and also how much a variation in +your input will affect your output. + +And that's all there is to it! Instead of `a -> b`'s, write `BVar s a -> BVar +s b`'s to compute what you want to know the gradient of. These are normal +functions, so you can use all of your favorite higher order functions and +combinators (like `(.)`, `map`, etc.). And once you're done, use `gradBP` to +compute that gradient. + +Note that `gradBP` requires a `Backprop` constraint on the input and output of +your function. `Backprop` is essentially the typeclass of values that can be +"backpropagated". For product types, this instance is automatically derivable. +But writing your own custom instances for your own types is also fairly +straightforward. More on this later! + +The rest of the package really is just ways to manipulate `BVar s a`s as if +they were just `a`s, to make everything as smooth as possible. Let's move on +to learning about **[ways to manipulate BVars][bvars]**! + +[bvars]: https://backprop.jle.im/03-manipulating-bvars.html diff --git a/doc/03-manipulating-bvars.md b/doc/03-manipulating-bvars.md new file mode 100644 --- /dev/null +++ b/doc/03-manipulating-bvars.md @@ -0,0 +1,430 @@ +--- +title: Manipulating BVars +--- + +Manipulating BVars +================== + +```haskell top hide +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DeriveGeneric #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE FlexibleInstances #-} +{-# LANGUAGE StandaloneDeriving #-} +{-# LANGUAGE TemplateHaskell #-} +{-# LANGUAGE TypeFamilies #-} +{-# LANGUAGE ViewPatterns #-} + + +import Data.Functor.Identity +import GHC.Generics (Generic) +import GHC.TypeNats +import Inliterate.Import +import Lens.Micro +import Lens.Micro.TH +import Numeric.Backprop +import Numeric.Backprop.Class +import Numeric.LinearAlgebra.Static (L, R) +import System.Random +import qualified Numeric.LinearAlgebra.Static as H +``` + +The most important aspect of the usability of this library is allowing you to +seamlessly manipulate `BVar s a`s as if they were just `a`s, without requiring +you as the user to be able to recognize or acknowledge the difference. Here +are some techniques to that end. + +Remember, a `BVar s a` is a `BVar` containing an `a` --- it's an `a` that, when +used, keeps track of and propagates your gradient. + +Typeclass Interface +------------------- + +`BVar`s have `Num`, `Fractional`, `Floating`, `Eq`, and `Ord` instances. These +instances are basically "lifted" to the `BVar` itself, so if you have a `BVar s +Double`, you can use `(*)`, `sqrt`, `(>)`, etc. on it exactly as if it were +just a `Double`. + +Constant Values +--------------- + +If we don't *care* about a value's gradient, we can use `auto`: + +```haskell +auto :: a -> BVar s a +``` + +`auto x` basically gives you a `BVar` that contains just `x` alone. Useful for +using with functions that expect `BVar`s, but you just have a specific value +you want to use. + +Coercible +--------- + +If `a` and `b` are `Coercible`, then so are `BVar s a` and `BVar s b`, using +the `coerceVar` function. This is useful for "unwrapping" and "wrapping" +`BVar`s of newtypes: + + +```haskell top +newtype MyInt = MyInt Int + +getMyInt :: BVar s MyInt -> BVar s Int +getMyInt = coerceVar +``` + +Accessing Contents +------------------ + +The following techniques can be used to access values inside `BVar`s: + +### Traversable Containers + +One that we saw earlier was `sequenceVar`, which we used to turn a `BVar` +containing a list into a list of `BVar`s: + +```haskell +sequenceVar :: (Backprop a, Reifies s W) + => BVar s [a] + -> [BVar s a] +``` + +If you have a `BVar` containing a list, you can get a list of `BVar`s of all of +that list's elements. (`sequenceVar` actually works on all `Traversable` +instances, not just lists) This is very useful when combined with +`-XViewPatterns`, as seen earlier. + +### Records and Fields + +In practice, a lot of usage involves functions involving contents of records or +data types containing fields. The previous example, involving a simple ANN, +demonstrates this: + +```haskell top +data Net = N { _nWeights1 :: L 20 100 + , _nBias1 :: R 20 + , _nWeights2 :: L 5 20 + , _nBias2 :: R 5 + } + deriving (Show, Generic) + +instance Backprop Net -- can be automatically defined +``` + +To compute the result of this network (ran on an `R 100`, a 100-vector) and get +the output `R 5`, we need do a matrix multiplication by the `_nWeights1` field, +add the result to the `_nBias1` field...basically, the result is a function of +linear algebra and related operations on the input and all of the contents of +the `Net` data type. However, you can't directly use `_nWeights`, since it +takes a `Net`, not `BVar s Net`. And you also can't directly pattern match on +the `N` constructor. + +There are two main options for this: the lens interface, and the higher-kinded +data interface. + +#### Lens Interface + +The most straightforward way to do this is the lens-based interface, using +`viewVar` or `^^.`. + +If we make lenses for `Net` using the *[lens][]* or *[microlens-th][]* packages: + +[lens]: http://hackage.haskell.org/package/lens +[microlens-th]: http://hackage.haskell.org/package/microlens-th + +```haskell top +makeLenses ''Net +``` + +Then `^.` from the *lens* or *[microlens][]* packages lets you retrieve a field +from a `Net`: + +[microlens]: http://hackage.haskell.org/package/microlens + +```haskell +(^. nWeights1) :: Net -> L 20 100 +(^. nBias1 ) :: Net -> R 20 +(^. nWeights2) :: Net -> L 5 20 +(^. nBias2 ) :: Net -> R 5 +``` + +And, `^^.` from *backprop* (also aliased as `viewVar`) lets you do the same +thing from a `BVar s Net` (a `BVar` containing your `Net`): + +```haskell +(^^. nWeights1) :: BVar s Net -> BVar s (L 20 100) +(^^. nBias1 ) :: BVar s Net -> BVar s (R 20) +(^^. nWeights2) :: BVar s Net -> BVar s (L 5 20) +(^^. nBias2 ) :: BVar s Net -> BVar s (R 5) +``` + +```haskell top hide +instance Backprop (R n) where + zero = zeroNum + add = addNum + one = oneNum + +instance (KnownNat n, KnownNat m) => Backprop (L n m) where + zero = zeroNum + add = addNum + one = oneNum + +(#>) + :: (KnownNat n, KnownNat m, Reifies s W) + => BVar s (L n m) -> BVar s (R m) -> BVar s (R n) +(#>) = liftOp2 . op2 $ \xs y -> + ( xs H.#> y + , \d -> (d `H.outer` y, H.tr xs H.#> d) + ) + +dot :: (KnownNat n, Reifies s W) => BVar s (R n) -> BVar s (R n) -> BVar s Double +dot = liftOp2 . op2 $ \x y -> + ( x `H.dot` y + , \d -> let d' = H.konst d + in (d' * y, x * d') + ) +``` + +With our lenses and `^^.`, we can write our network running function. This +time, I'll include the type! + +```haskell top +runNet :: Reifies s W + => BVar s Net + -> BVar s (R 100) + -> BVar s (R 5) +runNet net x = z + where + -- run first layer + y = logistic $ (net ^^. nWeights1) #> x + (net ^^. nBias1) + -- run second layer + z = logistic $ (net ^^. nWeights2) #> y + (net ^^. nBias2) + +logistic :: Floating a => a -> a +logistic x = 1 / (1 + exp (-x)) +``` + +Note that we are using versions of `#>` lifted for `BVar`s, from the +*[hmatrix-backprop][]* library: + +```haskell +(#>) :: BVar s (L m n) -> BVar s (R n) -> BVar s (R m) +``` + +[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop + +#### Higher-Kinded Data Interface + +Using the lens based interface, you can't directly pattern match and construct +fields. To allow for directly pattern matching, there's another interface +option involving the "Higher-Kinded Data" techniques described in [this +article][hkd]. + +[hkd]: http://reasonablypolymorphic.com/blog/higher-kinded-data/ + +If we had a type-family (that can be re-used for all of your data types): + +```haskell top +type family HKD f a where + HKD Identity a = a + HKD f a = f a +``` + +We can define `Net` instead as: + +```haskell top +data Met' f = M { _mWeights1 :: HKD f (L 20 100) + , _mBias1 :: HKD f (R 20) + , _mWeights2 :: HKD f (L 5 20) + , _mBias2 :: HKD f (R 5) + } + deriving Generic +``` + +Then our *original* type is: + +```haskell top +type Met = Met' Identity + +deriving instance Show Met +instance Backprop Met +``` + +`Met` is the same as `Net` in every way -- it can be pattern matched on to get +the `L 20 100`, etc. (the `Identity` disappears): + +```haskell top +getMetBias1 :: Met -> R 20 +getMetBias1 (M _ b _ _) = b +``` + +The benefit of this is that we can now directly pattern match on a `BVar s Met` +to get the internal fields as `BVar`s using `splitBV` as a view pattern (or the +`BV` pattern synonym): + +```haskell top +runMet :: Reifies s W + => BVar s Met + -> BVar s (R 100) + -> BVar s (R 5) +runMet (splitBV -> M w1 b1 w2 b2) x = z + where + -- run first layer + y = logistic $ w1 #> x + b1 + -- run second layer + z = logistic $ w2 #> y + b2 +``` + +Now, the `M w1 b1 w2 b2` pattern can be used to deconstruct *both* "normal" +`Met`s, as well as a `BVar s Met` (with `splitBV` or `BV`). + +### Potential or Many Fields + +Some values "may" or "may not" have values of a given field. An example would +include the nth item in a list or vector, or the `Just` of a `Maybe`. + +For these, the lens-based (prism-based/traversal-based) interface is the main way to access +partial fields. You can use `(^^?)` or `previewVar` with any `Traversal`: + +```haskell +(^?) :: a -> Traversal' a b -> Maybe b +(^^?) :: BVar s a -> Traversal' a b -> Maybe (BVar s b) +``` + +If the value in the `BVar` "has" that field, then you'll get a `Just` with the +`BVar` of that field's contents. If it doesn't, you'll get a `Nothing`. + +You can use this with any prism or traversal, like using `_head` to get the +first item in a list if it exists. + +If you have a type that might contain *many* values of a field (like a tree or +list), you can use `(^^..)` or `toListOfVar`, which works on any `Traversal`: + +```haskell +(^..) :: a -> Traversal' a b -> [ b] +(^^..) :: BVar s a -> Traversal' a b -> [BVar s b] +``` + +This can be used to implement `sequenceVar`, actually: + +```haskell +sequenceVar :: BVar s [a] -> [BVar s a] +sequenceVar xs = xs ^^.. traverse +``` + +### Tuples + +The `T2` pattern synonym is provided, which allow you to pattern match on a +`BVar s (a, b)` to get a `BVar s a` and `BVar s b`. The `T3` pattern is also +provided, which does the same thing for three-tuples. + +Note that `T2` and `T2` are bidirectional pattern synonyms, and can be used to +construct as well as deconstruct. + +Combining BVars +--------------- + +The following techniques can be used to "combine" `BVar`s: + +### Foldable Containers + +The "opposite" of `sequenceVar` is `collectVar`, which takes a foldable +container of `BVar`s and returns a `BVar` containing that foldable container of +contents: + +```haskell +collectVar :: (Backprop a, Foldable t, Functor t, Reifies s W) + => t (BVar s a) + -> BVar s (t a) +``` + +### Constructors + +Sometimes you would like to combine a bunch of `BVar`s into a `BVar` of +specific container or data type. + +#### isoVar + +The simplest way to do this is using the `isoVar`, `isoVar2`, etc. family of +functions: + +```haskell +isoVar2 + :: (Backprop a, Backprop b, Backprop c, Reifies s W) + => (a -> b -> c) + -> (c -> (a, b)) + -> BVar s a + -> BVar s b + -> BVar s c +``` + +So if we had a type like: + +```haskell top +data DoubleInt = DI Double Int +``` + +We can combine a `Double` and `Int` into a `DoubleInt` using `isoVar2`: + +```haskell +isoVar2 DI (\(DI x y) -> (x,y)) + :: Reifies s W + => BVar s Double + -> BVar s Int + -> BVar s DoubleInt +``` + +#### Higher-Kinded Data Interface + +You can also use the ["Higher Kinded Data"][hkd] interface, as well. For our +`Met` type above, you can use `joinBV`, or the `BV` pattern synonym: + +```haskell top +makeMet :: Reifies s W + => BVar s (L 20 100) + -> BVar s (R 20) + -> BVar s (L 5 20) + -> BVar s (R 5) + -> BVar s Met +makeMet w1 b1 w2 b2 = joinBV (M w1 b1 w2 b2) +``` + +### Modifying fields + +If you just want to "set" a specific field, you can use the lens-based +interface with `(.~~)` or `setVar`. For example, if we wanted to set the +`_nWeights2` field of a `Net` to a new matrix, we can do: + +```haskell +myNet & nWeights2 .~~ newMatrix +``` + +or + +```haskell +setVar nWeights2 + :: Reifies s W + => BVar s (L 20 5) + -> BVar s Net + -> BVar s Net +``` + +Prelude Modules +--------------- + +Finally, the *Prelude.Backprop* module has a lot of your normal Prelude +functions "lifted" to work on `BVar`s of values. For many situations, these +aren't necessary, and normal Prelude functions will work just fine on `BVar`s +of values (like `(.)`). However, it does have some convenient functions, like +`minimum`, `foldl'`, `fmap`, `toList`, `fromIntegral`, `realToFrac`, etc. +lifted to work on `BVar`s. This module is meant to be imported qualified. + +Moving On +========= + +Now that you know all about `BVar`s, you really can just **[jump into the +haddocks][haddock]** and start writing programs. The next section of this +documentation is more details about **[the `Backprop` typeclass][class]**. + +[class]: https://backprop.jle.im/04-the-backprop-typeclass.html +[haddock]: https://hackage.haskell.org/package/backprop diff --git a/doc/04-the-backprop-typeclass.md b/doc/04-the-backprop-typeclass.md new file mode 100644 --- /dev/null +++ b/doc/04-the-backprop-typeclass.md @@ -0,0 +1,132 @@ +--- +title: The Backprop Typeclass +--- + +The Backprop Typeclass +====================== + +```haskell top hide +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DeriveGeneric #-} + +import GHC.Generics (Generic) +import GHC.TypeNats +import Numeric.LinearAlgebra.Static (L, R) +import Numeric.Backprop +import Numeric.Backprop.Class +import qualified Data.Vector as V +``` + +Most of the functions in this module require a `Backprop` constraint on values +you wish to backpropagate. Even if you manage to get around it for the most +part, `gradBP` (the actual function to compute gradients) requires it on both +the inputs and outputs. Let's dig deeper into what it is, and how to define +instances. + +The Class +--------- + +The typeclass contains three methods: `zero`, `add`, and `one`: + +```haskell +class Backprop a where + zero :: a -> a + add :: a -> a -> a + one :: a -> a +``` + +`zero` is "zero" in the verb sense -- it takes a value and "zeroes out" all +components. For a vector, this means returning a zero vector of the same +shape. For a list, this means replacing all of the items with zero and +returning a list of the same length. `one` does the same thing but with one. +`add` is used to add together contributions in gradients, and is usually a +component-wise addition. + +Instances are provided for most common data types where it makes sense. + +Custom Instances +---------------- + +### Generics + +When defining your own custom types, if your custom type is has *a single +constructor* where all fields are instances of `Backprop`, then *GHC.Generics* +can be used to write your instances automatically: + +```haskell top hide +instance Backprop (R n) where + zero = zeroNum + add = addNum + one = oneNum + +instance (KnownNat n, KnownNat m) => Backprop (L n m) where + zero = zeroNum + add = addNum + one = oneNum +``` + +```haskell top +data MyType = MkMyType Double [Float] (R 10) (L 20 10) (V.Vector Double) + deriving Generic +``` + +Nice type. Since it has a single constructor and all of its fields are already +`Backprop` instances, we can just write: + +```haskell top +instance Backprop MyType +``` + +and now your type can be backpropagated! + +### Common Patterns + +For writing "primitive" `Backprop` instances (types that aren't product types), +you can use the provided "helpers" from the *Numeric.Backprop.Class* module. + +If your type is a `Num` instance, you can use `zeroNum`, `addNum`, and +`oneNum`: + +```haskell +instance Backprop Double where + zero = zeroNum + add = addNum + one = oneNum +``` + +If your type is made using a `Functor` instance, you can use `zeroFunctor` and +`oneFunctor`: + +```haskell +instance Backprop a => Backprop (V.Vector a) where + zero = zeroFunctor + add = undefined -- ?? + one = oneFunctor +``` + +And if your type has an `IsList` instance, you can use `addIsList`: + +```haskell +instance Backprop a => Backprop (V.Vector a) where + zero = zeroFunctor + add = addIsList + one = oneFunctor +``` + +### Completely Custom + +Completely custom instances are also possible; you just need to implement +`zero`, `add`, and `one` as they make sense for your type. Just make sure that +you obey [the laws][laws] for sane behavior! + +[laws]: http://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Class.html + +Moving On +========= + +At this point, feel free to **[jump into the haddocks][haddock]**, or read on +further for **[a list of applications and resources][applications]**. + +[haddock]: https://hackage.haskell.org/package/backprop +[applications]: https://backprop.jle.im/05-applications.html diff --git a/doc/05-applications.md b/doc/05-applications.md new file mode 100644 --- /dev/null +++ b/doc/05-applications.md @@ -0,0 +1,48 @@ +--- +title: Applications and Resources +--- + +Applications and Resources +========================== + +Congratulations! You are now a *backprop* master. Maybe you've even looked at +the [haddocks][haddock], which has the technical run-down of all of the +functions and types in this library. Now what? + +* Check out my [Introducing the backprop library][intro] blog post where I + announced the library to the world. In it, I introduce the library by + building and training a full artificial neural network with it, and use it + to classify the famous MNIST handwritten digit data set. + +* If you want an even more high-level perspective and inspiration, check out + my [A Purely Functional Typed Approach to Trainable Models][models] blog + series, where I talk about how looking at modeling through the lens of + differentiable programming with purely functional typed code can provide + new insights and help you develop and train effective models. + +* While they are mostly re-phrasings of the two things above, I also have + some [example projects as literate haskell files][lhs] on the github + repository for the library. These are also [rendered as pdfs][renders] for + easier reading. + +* If you're doing anything with linear algebra, why not check out the + *[hmatrix-backprop][]* library, which provides the "backprop-lifted" + operations that all of the above examples rely on for linear algebra + operations? + +[haddock]: https://hackage.haskell.org/package/backprop +[intro]: https://blog.jle.im/entry/introducing-the-backprop-library.html +[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html +[lhs]: https://github.com/mstksg/backprop/blob/master/samples +[renders]: https://github.com/mstksg/backprop/tree/master/renders +[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop + +This is the end of the "end-user" documentation for *backprop*! The rest of +all you need to know to use the library is in the **[haddocks on +hackage][haddock]**. + +However, if you are a library writer who wants to offer your users the ability +to backpropagate your library functions, let's move on to the **[library +maintainer's guide to equipping your library with backprop][equipping]**! + +[equipping]: https://backprop.jle.im/06-equipping-your-library.html diff --git a/doc/06-equipping-your-library.md b/doc/06-equipping-your-library.md new file mode 100644 --- /dev/null +++ b/doc/06-equipping-your-library.md @@ -0,0 +1,776 @@ +--- +title: Equipping your Library +--- + +Equipping your Library for Backprop +=================================== + +```haskell top hide +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DeriveGeneric #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE FlexibleInstances #-} +{-# LANGUAGE StandaloneDeriving #-} +{-# LANGUAGE TemplateHaskell #-} +{-# LANGUAGE TypeFamilies #-} +{-# LANGUAGE ViewPatterns #-} + + +import Data.Functor.Identity +import qualified Data.List +import GHC.Generics (Generic) +import GHC.TypeNats +import Inliterate.Import +import Lens.Micro +import Lens.Micro.TH +import Numeric.Backprop +import Numeric.Backprop.Class +import Numeric.LinearAlgebra.Static (L, R, konst) +import System.Random +import qualified Data.Vector as V +import qualified Numeric.LinearAlgebra.Static as H +import qualified Numeric.LinearAlgebra as HU +``` + +So you want your users to be able to use your numerical library with +*backprop*, huh? + +This page is specifically for library authors who want to allow their users to +use their library operations and API with *backprop*. End-users of the +*backprop* library should not have to worry about the contents of this page. + +Equipping your library with backprop involves providing "backprop-aware" +versions of your library functions. *In fact*, it is possible to make a +library fully by providing *only* backprop versions of your functions, since +you can use a backprop-aware function as a normal function with `evalBP`. +Alternatively, you can re-export all of your functions in a separate module with +"backprop-aware" versions. + +Know Thy Types +-------------- + +The most significant effort will be in lifting your library's functions. If +you have a function: + +```haskell +myFunc :: a -> b +``` + +Then its lifted version would have type: + +```haskell +myFunc :: Reifies s W => BVar s a -> BVar s b +``` + +That is, instead of a function directly taking an `a` and returning a `b`, it's +a function taking a `BVar` containing an `a`, and returning a `BVar` containing +a `b`. + +Functions taking multiple arguments can be translated pretty straightforwardly: + +```haskell +func1 :: a -> b -> c +func1BP :: Reifies s W => BVar s a -> BVar s b -> BVar s c +``` + +And also functions returning multiple arguments: + +```haskell +func2 :: a -> ( b, c) +func2BP :: Reifies s W => BVar s a -> (BVar s b, BVar s c) +``` + +It is recommended (for ease of use with `-XTypeApplications`) that `Reifies s +W` be the *final* constraint in all code you write. + +Note that almost all operations involving `BVar`'d items require that the +contents have a `Backprop` instance. Alternative API's to backprop that +require `Num` instances instead (or explicitly specified addition functions) +are available in *Numeric.Backprop.Num* and *Numeric.Backprop.Explicit*. + +The Easy Way +------------ + +`BVar` based functions are just normal functions, so they can be applied +normally and passed as first-class values. If possible, if you can *utilize* +functions that are already `BVar`'d/lifted, then you can just define your API +in terms of those lifted functions. This is also how *users* are expected to +be able to use your library: just use the lifted functions you provide, in +order to make their own lifted functions using normal function application and +composition. + +However, if no lifted primitive functions are available, then you do have to do +some legwork to provide information on gradient computation for your types. +Ideally, you would only need to do this for some minimal set of your +operations, and then define the rest of them in terms of the functions you have +already lifted. + +Lifting operations manually +--------------------------- + +A `BVar s a -> BVar s b` really encodes two things: + +1. A `a -> b` (the actual function) +2. A `a -> b -> a` (the "scaled gradient" function) + +The documentation for [Numeric.Backprop.Op][op] gives detail about what these +entail, with rendered math and examples. + +[op]: http://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Op.html + +The second function requires some elaboration. Let's say you are writing a +lifted version of your function \\(y = f(x)\\) (whose derivative is +\\(\frac{dy}{dx}\\)), and that your *final result* at the end of your computation +is \\(z = g(f(x))\\) (whose derivative is \\(\frac{dz}{dx}\\)). In that case, because of the +chain rule, \\(\frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx}\\). + +The scaled gradient is the function which, *given* \\(\frac{dy}{dz}\\), *returns* +\\(\frac{dz}{dx}\\). (that is, returns \\(\frac{dz}{dy} \frac{dy}{dx}\\)). + +For example, for the mathematical operation \\(y = f(x) = x^2\\), then, considering +\\(z = g(f(x))\\), \\(\frac{dz}{dx} = \frac{dz}{dy} 2x\\). +In fact, for all functions taking and returning scalars (just normal single +numbers), \\(\frac{dz}{dx} = \frac{dz}{dy} f'(x)\\). + +With that in mind, let's write our "squared" op: + +```haskell top +square + :: (Num a, Backprop a, Reifies s W) + => BVar s a + -> BVar s a +square = liftOp1 . op1 $ \x -> + ( x^2 , \dzdy -> dzdy * 2 * x) +-- ^- actual result ^- scaled gradient function +``` + +Keeping along the same pattern, for \\(y = f(x) = \sin(x)\\), then, considering \\(z += g(f(x))\\), \\(\frac{dz}{dx} = \frac{dz}{dy} \cos(x)\\). So, we have: + +```haskell top +liftedSin + :: (Floating a, Backprop a, Reifies s W) + => BVar s a + -> BVar s a +liftedSin = liftOp1 . op1 $ \x -> + ( sin x, \dzdy -> dzdy * cos x ) +``` + +In general, for functions that take and return scalars: + +```haskell +liftedF + :: (Reifies s W, Backprop a, Num a) + => BVar s a + -> BVar s a +liftedF = liftOp1 . op1 $ \x -> + ( f x, \dzdy -> dzdy * dfdx x ) +``` + +For an example of every single numeric function in base Haskell, see [the +source of Op.hs][opsource] for the `Op` definitions for every method in `Num`, +`Fractional`, and `Floating`. + +[opsource]: https://github.com/mstksg/backprop/blob/a7651b4549048a3aca73c79c6fbe07c3e8ee500e/src/Numeric/Backprop/Op.hs#L646-L787 + +### Non-trivial example + +A simple non-trivial example is `sumElements`, which we can define to take the +*hmatrix* library's `R n` type (an n-vector of `Double`). In this case, we +have to think about \\(g(\mathrm{sum}(\mathbf{x}))\\). In this case, the types +guide our thinking: + +```haskell +sumElements :: R n -> Double +sumElementsScaledGrad :: R n -> Double -> R n +``` + +The simplest way for me to do this personally is to just take it element by +element. + +1. *Write out the functions in question, in a simple example* + + In our case: + + * \\(y = f(\langle a, b, c \rangle) = a + b + c\\) + * \\(z = g(y) = g(a + b + c)\\) + +2. *Identify the components in your gradient* + + In our case, we have to return a gradient \\(\langle \frac{\partial z}{\partial a}, + \frac{\partial z}{\partial b}, \frac{\partial z}{\partial c} \rangle\\). + +3. *Work out each component of the gradient until you start to notice a + pattern* + + Let's start with \\(\frac{\partial z}{\partial a}\\). We need to find + \\(\frac{\partial z}{\partial a}\\) in terms of \\(\frac{dz}{dy}\\): + + * Through the chain rule, \\(\frac{\partial z}{\partial a} = + \frac{dz}{dy} \frac{\partial y}{\partial a}\\). + * Because \\(y = a + b + c\\), we know that \\(\frac{\partial y}{\partial + a} = 1\\). + * Because \\(\frac{\partial y}{\partial a} = 1\\), we know that + \\(\frac{\partial z}{\partial a} = \frac{dz}{dy} \times 1 = + \frac{dz}{dy}\\). + + So, our expression of \\(\frac{\partial z}{\partial a}\\) in terms of + \\(\frac{dz}{dy}\\) is simple -- it's simply \\(\frac{\partial z}{\partial + a} = \frac{dz}{dy}\\). + + Now, let's look at \\(\frac{\partial z}{\partial b}\\). We need to find + \\(\frac{\partial z}{\partial b}\\) in terms of \\(\frac{dz}{dy}\\). + + * Through the chain rule, \\(\frac{\partial z}{\partial b} = + \frac{dz}{dy} \frac{\partial y}{\partial b}\\). + * Because \\(y = a + b + c\\), we know that \\(\frac{\partial y}{\partial + b} = 1\\). + * Because \\(\frac{\partial y}{\partial b} = 1\\), we know that + \\(\frac{\partial z}{\partial b} = \frac{dz}{dy} \times 1 = + \frac{dz}{dy}\\). + + It looks like \\(\frac{\partial z}{\partial b} = \frac{\partial z}{\partial + y}\\), as well. + + At this point, we start to notice a pattern. We can apply the same logic + to see that \\(\frac{\partial z}{\partial c} = \frac{dz}{dy}\\). + +4. *Write out the pattern* + + Extrapolating the pattern, \\(\frac{\partial z}{\partial q}\\), where + \\(q\\) is *any* component, is always going to be a constant -- + \\(\frac{dz}{dy}\\). + +So in the end: + +```haskell top hide +instance Backprop (R n) where + zero = zeroNum + add = addNum + one = oneNum + +instance (KnownNat n, KnownNat m) => Backprop (L n m) where + zero = zeroNum + add = addNum + one = oneNum + +sumElements :: KnownNat n => R n -> Double +sumElements = HU.sumElements . H.extract +``` + +```haskell top +liftedSumElements + :: (KnownNat n, Reifies s W) + => BVar s (R n) + -> BVar s Double +liftedSumElements = liftOp1 . op1 $ \xs -> + ( sumElements xs, \dzdy -> konst dzdy ) -- a constant vector +``` + +### Multiple-argument functions + +Lifting multiple-argument functions is the same thing, except using `liftOp2` +and `op2`, or `liftOpN` and `opN`. + +A `BVar s a -> BVar s b -> BVar s c` encodes two things: + +1. The actual `a -> b -> c` +2. The scaled gradient, `a -> b -> c -> (a, b)`. + +The `c` parameter of the scaled gradient is again \\(\frac{dz}{dy}\\), and the +final `(a,b)` is a tuple of \\(\frac{\partial z}{\partial x_1}\\) and +\\(\frac{\partial z}{\partial x_2}\\): how \\(\frac{dz}{dy}\\) affects both of +the inputs. + +For a simple example, let's look at \\(x + y\\). Working it out: + +* \\(y = f(x_1, x_2) = x_1 + x_2\\) +* \\(z = g(f(x_1, x_2)) = g(x_1 + x_2)\\) +* Looking first for \\(\frac{\partial z}{\partial x_1}\\) in terms of + \\(\frac{dz}{dy}\\): + * \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial + y}{\partial x_1}\\) (chain rule) + * From \\(y = x_1 + x_2\\), we see that \\(\frac{\partial y}{\partial + x_1} = 1\\) + * Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \times 1 + = \frac{dz}{dy}\\). +* Looking second for \\(\frac{\partial z}{\partial x_2}\\) in terms of + \\(\frac{dz}{dy}\\): + * \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} \frac{\partial + y}{\partial x_2}\\) (chain rule) + * From \\(y = x_1 + x_2\\), we see that \\(\frac{\partial y}{\partial + x_2} = 1\\) + * Therefore, \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} \times 1 + = \frac{dz}{dy}\\). +* Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy}\\), and also + \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy}\\). + +Putting it into code: + +```haskell top +add :: (Num a, Backprop a, Reifies s W) + => BVar s a + -> BVar s a + -> BVar s a +add = liftOp2 . op2 $ \x1 x2 -> + ( x1 + x2, \dzdy -> (dzdy, dzdy) ) +``` + +Let's try our hand at multiplication, or \\(x * y\\): + +* \\(y = f(x_1, x_2) = x_1 x_2\\) +* \\(z = g(f(x_1, x_2)) = g(x_1 x_2)\\) +* Looking first for \\(\frac{d\partial }{d\partial _1}\\) in terms of + \\(\frac{dz}{dy}\\): + * \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial + y}{\partial x_1}\\) (chain rule) + * From \\(y = x_1 x_2\\), we see that \\(\frac{\partial y}{\partial x_1} + = x_2\\) + * Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} x_2\\). +* Looking second for \\(\frac{\partial z}{\partial x_2}\\) in terms of + \\(\frac{dz}{dy}\\): + * \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial + y}{\partial x_1}\\) (chain rule) + * From \\(y = x_1 x_2\\), we see that \\(\frac{\partial y}{\partial x_2} + = x_1\\) + * Therefore, \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} x_1\\). +* Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} x_2\\), and + \\(\frac{\partial z}{\partial x_2} = x_1 \frac{dz}{dy}\\). + +In code: + +```haskell top +mul :: (Num a, Backprop a, Reifies s W) + => BVar s a + -> BVar s a + -> BVar s a +mul = liftOp2 . op2 $ \x1 x2 -> + ( x1 * x2, \dzdy -> (dzdy * x2, x1 * dzdy) ) +``` + +For non-trivial examples involving linear algebra, see the source for the *[hmatrix-backprop][]* library. + +[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop + +Some examples, for the dot product between two vectors and for matrix-vector +multiplication: + +```haskell top +-- import qualified Numeric.LinearAlgebra.Static as H + +-- | dot product between two vectors +dot + :: (KnownNat n, Reifies s W) + => BVar s (R n) + -> BVar s (R n) + -> BVar s Double +dot = liftOp2 . op2 $ \u v -> + ( u `H.dot` v + , \dzdy -> (H.konst dzdy * v, u * H.konst dzdy) + ) + + +-- | matrix-vector multiplication +(#>) + :: (KnownNat m, KnownNat n, Reifies s W) + => BVar s (L m n) + -> BVar s (R n) + -> BVar s (R m) +(#>) = liftOp2 . op2 $ \mat vec -> + ( mat H.#> vec + , \dzdy -> (dzdy `H.outer` vec, H.tr mat H.#> dzdy) + ) +``` + +### Returning multiple items + +You can return tuples inside `BVar`s: + +```haskell top +splitAt + :: (Backprop a, Reifies s W) + => Int + -> BVar s [a] + -> BVar s ([a], [a]) +splitAt n = liftOp1 . op1 $ \xs -> + let (ys, zs) = Data.List.splitAt n xs + in ((ys, zs), \(dys,dzs) -> dys ++ dzs) + -- assumes dys and dzs have the same lengths as ys and zs +``` + +This works as expected. However, it is recommended, for the benefit of your +users, that you return a tuple of `BVar`s instead of a `BVar` of tuples: + +```haskell top +splitAt' + :: (Backprop a, Reifies s W) + => Int + -> BVar s [a] + -> (BVar s [a], BVar s [a]) +splitAt' n xs = (yszs ^^. _1, yszs ^^. _2) + where + yszs = liftOp1 (op1 $ \xs' -> + let (ys, zs) = Data.List.splitAt n xs' + in ((ys, zs), \(dys,dzs) -> dys ++ dzs) + ) xs +``` + +using `_1` and `_2` from the *[microlens][]* or *[lens][]* packages. This +might also be cleaner if you take advantage of the `T2` or `T3` pattern +synonyms: + +[microlens]: http://hackage.haskell.org/package/microlens +[lens]: http://hackage.haskell.org/package/lens + +```haskell top +splitAt'' + :: (Backprop a, Reifies s W) + => Int + -> BVar s [a] + -> (BVar s [a], BVar s [a]) +splitAt'' n xs = (ys, zs) + where + T2 ys zs = liftOp1 (op1 $ \xs' -> + let (ys, zs) = Data.List.splitAt n xs' + in ((ys, zs), \(dys,dzs) -> dys ++ dzs) + ) xs +``` + +### Isomorphisms + +If your function witnesses an isomorphism, there are handy combinators for +making this easy to write. This is especially useful in the case of data +constructors: + +```haskell top +newtype Foo = MkFoo { getFoo :: Double } + deriving Generic + +instance Backprop Foo + +mkFoo + :: Reifies s W + => BVar s Double + -> BVar s Foo +mkFoo = isoVar MkFoo getFoo + +data Bar = MkBar { bar1 :: Double, bar2 :: Float } + deriving Generic + +instance Backprop Bar + +mkBar + :: Reifies s W + => BVar s Double + -> BVar s Float + -> BVar s Bar +mkBar = isoVar2 MkBar (\b -> (bar1 b, bar2 b)) +``` + +Note also that if you have a newtype with one constructor (or any other two +`Coercible` types), you can simply use `coerceVar`: + +```haskell top +mkFoo' + :: BVar s Double + -> BVar s Foo +mkFoo' = coerceVar -- requires no `Reifies s W` constraint +``` + +### NoGrad + +If you do decide to go to the extreme, and provide *only* a BVar-based +interface to your library (and no non-BVar based one), then you might have a +situation where you have a function where you cannot define the gradient -- +maybe no gradient exists, or you haven't put in the time to write one. In this +case, you can use `noGrad` and `noGrad1`: + +```haskell top +negateNoGrad + :: (Num a, Backprop a, Reifies s W) + => BVar s a + -> BVar s a +negateNoGrad = liftOp1 (noGrad1 negate) +``` + +This function can still be used with `evalBP` to get the correct answer. It +can even be used with `gradBP` if the result is never used in the final answer. + +However, if it *is* used in the final answer, then computing the gradient will +throw a runtime exception. + +Be sure to warn your users! Like any partial function, this is not recommended +unless in extreme circumstances. + +Monadic Operations +------------------ + +This should all work if your operations are all "pure". However, what about +the cases where your operations have to be performed in some Applicative or +Monadic context? + +For example, what if `add :: X -> X -> IO X` ? + +One option you can do is to newtype-wrap your operations, and then give those a +backprop instance: + +```haskell top hide +data X + +zeroForX :: X -> X +zeroForX = undefined +addForX :: X -> X -> IO X +addForX = undefined +oneForX :: X -> X +oneForX = undefined +``` + +```haskell top +newtype IOX = IOX (IO X) + +instance Backprop IOX where + zero (IOX x) = IOX (fmap zeroForX x) + -- or, depending on the type of `zeroForX`: + -- zero (IOX x) = IOX (zeroForX =<< x) + + add (IOX x) (IOX y) = IOX $ do + x' <- x + y' <- y + addForX x' y' + + one (IOX x) = IOX (fmap oneForX x) +``` + +And you can define your functions in terms of this: + +```haskell top +addX + :: Reifies s W + => BVar s IOX + -> BVar s IOX + -> BVar s IOX +addX = liftOp2 . op2 $ \(IOX x) (IOX y) -> + ( IOX (do x' <- x; y' <- y; addForX x' y') + , \dzdy -> (dzdy, dzdy) + ) +``` + +This should work fine as long as you never "branch" on any *results* of your +actions. You must not ever need to peek inside the *results* of the action in +order to decide *what* operations to do next. In other words, this works if +the operations you need to perform are all known and fixed before-hand, before +any actions are performed. So, this means no access to the `Eq` or `Ord` +instances of BVars (unless your monad has `Eq` or `Ord` instances defined). + +A newtype wrapper is provided to give you this behavior automatically -- it's +`ABP`, from *Numeric.Backprop* and *Numeric.Backprop.Class*. + +```haskell +type IOX = ABP IO X +``` + +However, this will not work if you need to do things like compare contents, +etc. to decide what operations to use. + +At the moment, this is not supported. Please open an issue if this becomes an +issue! + +Supporting Data Types +--------------------- + +Your library will probably have data types that you expect your users to use. +To equip your data types for backpropagation, you can take a few steps. + +### Backprop Class + +First of all, all of your library's types should have instances of the +[`Backprop` typeclass][class]. This allows values of your type to be used in +backpropagatable functions. See the [Backprop typeclass section][tcdocs] of +this documentation for more information on writing a `Backprop` instance for +your types. + +[class]: https://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Class.html +[tcdocs]: https://backprop.jle.im/04-the-backprop-typeclass.html + +In short: + +1. If your type is a type with a single constructor whose fields are all + instances of `Backprop`, you can just write `instance Backprop MyType`, and + the instance is generated automatically (as long as your type has a + `Generic` instance) + + ```haskell top + data MyType = MkMyType Double [Float] (R 10) (L 20 10) (V.Vector Double) + deriving Generic + + instance Backprop MyType + ``` + +2. If your type is an instance of `Num`, you can use `zeroNum`, `addNum`, and + `oneNum` to get free definitions of the typeclass methods. + + ```haskell + instance Backprop Double where + zero = zeroNum + add = addNum + one = oneNum + ``` + +3. If your type is made using a `Functor` instance, you can use `zeroFunctor` + and `oneFunctor`: + + ```haskell + instance Backprop a => Backprop (V.Vector a) where + zero = zeroFunctor + add = undefined -- ?? + one = oneFunctor + ``` + +4. If your type has an `IsList` instance, you can use `addIsList`: + + ```haskell + instance Backprop a => Backprop (V.Vector a) where + zero = zeroFunctor + add = addIsList + one = oneFunctor + ``` + +For more details, see the [aforementioned documentation][tcdocs] or the [actual +typeclass haddock documentation][class]. + +### Accessors + +If you have product types, users should be able to access values inside `BVar`s +of your data type. There are two main ways to provide access: the lens-based +interface and the higher-kinded-data-based interface. + +The lens-based interface gives your users "getter" and "setter" functions for +fields, and the higher-kinded-data-based interface lets your users pattern +match on your data type's original constructor to get fields and construct +values. + +#### Lens-Based Interface + +If you are defining a product type, like + +```haskell top +data MyType = MT { _mtDouble :: Double + , _mtInt :: Int + , _mtDoubles :: [Double] + } +``` + +Users who have a `BVar s MyType` can't normally access the fields inside, +because you can't directly pattern match normally, and the record accessors +are `MyType -> Int` (unlifted). As a library maintainer, you can provide them +*lenses* to the fields, either generated automatically using the *[lens][]* or +*[microlens-th][]* packages: + +[lens]: http://hackage.haskell.org/package/lens +[microlens-th]: http://hackage.haskell.org/package/microlens-th + +```haskell top +makeLenses ''MyType +``` + +or manually by hand: + +```haskell top +mtInt' :: Functor f => (Int -> f Int) -> MyType -> f MyType +mtInt' f mt = (\i -> mt { _mtInt = i }) <$> f (_mtInt mt) +``` + +Now, users can use `^.` or `view` from the *lens* or *[microlens][]* packages +to retrieve your fields: + +[microlens]: http://hackage.haskell.org/package/microlens + +```haskell +(^. mtDouble) :: MyType -> Double +``` + +And `(^^.)` and `viewVar` from *backprop* to retrieve fields from a `BVar`: + +```haskell +(^^. mtDouble) :: BVar s MyType -> BVar s Double +``` + +They can also use `set` or `.~` to modify fields, and `setVar` and `.~~` to +modify and "set" fields in a `BVar`: + +```haskell +set mtDouble :: Double -> MyType -> MyType +setVar mtDouble :: BVar s Double -> BVar s MyType -> BVar s MyType +``` + +#### Higher-Kinded Data Interface + +The alternative "Higher-Kinded Data" technique, inspired by [this +article][hkd], allows your users to directly pattern match on `BVar`s of your +types to get their contents. + +[hkd]: http://reasonablypolymorphic.com/blog/higher-kinded-data/ + +Doing this requires modifying the definition of your data types slightly. +Instead of `MyType` above, we can make a type family that can be re-used for +all of your data types: + +```haskell top +type family HKD f a where + HKD Identity a = a + HKD f a = f a +``` + +and define your data types in terms of this type family (remembering to derive +`Generic`): + +```haskell top +data MyType2' f = MT2 { mt2Double :: HKD f Double + , mt2Int :: HKD f Int + , mt2Doubles :: HKD f [Double] + } + deriving Generic +``` + +Now your original data type can be recovered with `MyType2' Identity`, and can +be pattern matched directly in the same way as the original type (the +`Identity` disappears): + +```haskell top +type MyType2 = MyType2' Identity + +deriving instance Show MyType2 +instance Backprop MyType2 + +getMT2Double :: MyType2 -> Double +getMT2Double (MT2 d _ _) = d +``` + +But now, users can *pattern match* on a `BVar s MyType2` to get `BVar`s of the +contents, with `splitBV` or the `BV` pattern synonym: + +```haskell top +getMT2DoubleBVar + :: Reifies s W + => BVar s MyType2 + -> BVar s Double +getMT2DoubleBVar (splitBV -> MT2 d _ _) = d +``` + +Under `splitBV`, your users can pattern match on the `MT2` constructor and get +the contents as `BVar`s. + +Users can also use `joinBV` (or the `BV` pattern synonym in constructor mode) +to re-construct a `BVar` of `MyType2` in terms of `BVar`s of its contents using +the `MT2` constructor: + +```haskell top +makeMyType2 + :: Reifies s W + => BVar s Double + -> BVar s Int + -> BVar s [Double] + -> BVar s MyType2 +makeMyType2 d i ds = joinBV $ MT2 d i ds +``` + diff --git a/doc/index.md b/doc/index.md new file mode 100644 --- /dev/null +++ b/doc/index.md @@ -0,0 +1,61 @@ +--- +title: Home +--- + +Welcome to Backprop +=================== + +Automatic *heterogeneous* back-propagation. + +*Write your functions normally* to compute your result, and the library will +*automatically compute your gradient*! + +```haskell top hide +import Numeric.Backprop +``` + +```haskell eval +gradBP (\x -> x^2 + 3) (9 :: Double) +``` + +Differs from [ad][] by offering full heterogeneity -- each intermediate step +and the resulting value can have different types (matrices, vectors, scalars, +lists, etc.) + +[ad]: http://hackage.haskell.org/package/ad + +```haskell eval +gradBP2 (\x xs -> sum (map (**2) (sequenceVar xs)) / x) + (9 :: Double ) + ([1,6,2] :: [Double]) +``` + +Useful for applications in [differential programming][dp] and deep learning for +creating and training numerical models, especially as described in this blog +post on [a purely functional typed approach to trainable models][models]. +Overall, intended for the implementation of gradient descent and other numeric +optimization techniques. Comparable to the python library [autograd][]. + +[dp]: https://www.facebook.com/yann.lecun/posts/10155003011462143 +[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html +[autograd]: https://github.com/HIPS/autograd + +**[Get started][getting started]** with the introduction and walkthrough! Full +technical documentation is also **[available on hackage][hackage]** if you want +to skip the introduction and get right into using the library. Support is +available on the **[gitter channel][gitter]**! + +[getting started]: https://backprop.jle.im/01-getting-started.html + +[hackage]: http://hackage.haskell.org/package/backprop +[gitter]: https://gitter.im/haskell-backprop/Lobby + +[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) + +[![Beerpay](https://beerpay.io/mstksg/backprop/badge.svg?style=beer-square)](https://beerpay.io/mstksg/backprop) + +[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=86400)](https://hackage.haskell.org/package/backprop) +[![backprop on Stackage LTS 11](http://stackage.org/package/backprop/badge/lts-11)](http://stackage.org/lts-11/package/backprop) +[![backprop on Stackage Nightly](http://stackage.org/package/backprop/badge/nightly)](http://stackage.org/nightly/package/backprop) +[![Build Status](https://travis-ci.org/mstksg/backprop.svg?branch=master)](https://travis-ci.org/mstksg/backprop) + diff --git a/src/Numeric/Backprop.hs b/src/Numeric/Backprop.hs --- a/src/Numeric/Backprop.hs +++ b/src/Numeric/Backprop.hs @@ -1,3 +1,4 @@ +{-# LANGUAGE CPP #-} {-# LANGUAGE DataKinds #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs #-} @@ -49,8 +50,8 @@ -- reading the docs for 'BVar'. -- -- If you are writing a library, see --- --- for a guide for equipping your library with backpropatable operations. +-- for a guide for +-- equipping your library with backpropatable operations. -- -- In the original version 0.1, this module required 'Num' instances for -- methods instead of 'Backprop' instances. This interface is still @@ -90,6 +91,7 @@ -- $hkd , splitBV , joinBV + , pattern BV , E.BVGroup -- * 'Op' , Op(..) @@ -111,6 +113,7 @@ , Reifies ) where +import Data.Bifunctor import Data.Maybe import Data.Reflection import Data.Type.Index @@ -195,20 +198,21 @@ => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as -> (b, Tuple as) -backpropN = E.backpropN E.zeroFuncs E.oneFunc +backpropN f = second ($ E.oneFunc) . E.backpropN E.zeroFuncs f {-# INLINE backpropN #-} -- | 'backpropN', but allows you to provide the gradient of the "final -- result" with respect to the output of your function. See 'backpropWith' -- for more details. +-- +-- Note that argument order changed in v0.2.3. -- -- @since 0.2.0.0 backpropWithN :: (Every Backprop as, Known Length as) => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, Tuple as) + -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function backpropWithN = E.backpropWithN E.zeroFuncs {-# INLINE backpropWithN #-} @@ -224,7 +228,7 @@ => (forall s. Reifies s W => BVar s a -> BVar s b) -> a -> (b, a) -backprop = E.backprop E.zeroFunc E.oneFunc +backprop f = second ($ E.oneFunc) . E.backprop E.zeroFunc f {-# INLINE backprop #-} -- | A version of 'backprop' that allows you to specify the gradent of your @@ -241,13 +245,14 @@ -- 'backprop' is essentially 'backpropWith' with @'const' 1@ for scalars -- and 'Num' instances. -- +-- Note that argument order changed in v0.2.3 +-- -- @since 0.2.0.0 backpropWith :: Backprop a => (forall s. Reifies s W => BVar s a -> BVar s b) -> a - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, a) + -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function backpropWith = E.backpropWith E.zeroFunc {-# INLINE backpropWith #-} @@ -296,21 +301,22 @@ -> a -> b -> (c, (a, b)) -backprop2 = E.backprop2 E.zeroFunc E.zeroFunc E.oneFunc +backprop2 f x = second ($ E.oneFunc) . E.backprop2 E.zeroFunc E.zeroFunc f x {-# INLINE backprop2 #-} -- | 'backprop2', but allows you to provide the gradient of the "final -- result" with respect to the output of your function. See 'backpropWith' -- for more details. -- +-- Note that argument order changed in v0.2.3 +-- -- @since 0.2.0.0 backpropWith2 :: (Backprop a, Backprop b) => (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c) -> a -> b - -> (c -> c) -- ^ Gradient of final result with respect to output of function - -> (c, (a, b)) + -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function backpropWith2 = E.backpropWith2 E.zeroFunc E.zeroFunc {-# INLINE backpropWith2 #-} @@ -559,11 +565,13 @@ -- gradient is assumed to correspond with the second item in the input, -- etc.; this can cause unexpected behavior in 'Foldable' instances that -- don't have a fixed number of items. +-- +-- Prior to v0.2.3, required a 'Backprop' constraint on @t a@. collectVar - :: (Foldable t, Functor t, Backprop a, Backprop (t a), Reifies s W) + :: (Foldable t, Functor t, Backprop a, Reifies s W) => t (BVar s a) -> BVar s (t a) -collectVar = E.collectVar E.addFunc E.zeroFunc E.zeroFunc +collectVar = E.collectVar E.addFunc E.zeroFunc {-# INLINE collectVar #-} -- | Lift an 'Op' with an arbitrary number of inputs to a function on the @@ -715,7 +723,9 @@ pattern T2 x y <- (\xy -> (xy ^^. _1, xy ^^. _2) -> (x, y)) where T2 = isoVar2 (,) id -{-# COMPLETE T2 #-} +#if MIN_VERSION_base(4,10,0) +{-# COMPLETE BV #-} +#endif -- | Useful pattern for constructing and deconstructing 'BVar's -- three-tuples. @@ -730,7 +740,9 @@ pattern T3 x y z <- (\xyz -> (xyz ^^. _1, xyz ^^. _2, xyz ^^. _3) -> (x, y, z)) where T3 = isoVar3 (,,) id -{-# COMPLETE T3 #-} +#if MIN_VERSION_base(4,10,0) +{-# COMPLETE BV #-} +#endif -- $hkd -- @@ -786,6 +798,13 @@ -- myFunction ('splitBV' -> MT x y) = x + 'Prelude.Backprop.sum' y -- @ -- +-- Or also, using the 'BV' pattern synonym: +-- +-- @ +-- myFunction :: 'BVar' s MyType -> BVar s Double +-- myFunction ('BV' (MT x y)) = x + 'Prelude.Backprop.sum' y +-- @ +-- -- If you use 'splitBV', the contents will be a @BVar s Double@ and a @BVar -- s [Double]@. It lets you "extract" the fields, because your 'MyType'' -- constructor now holds a @'BVar' s Double@ and a @BVar s [Double]@, @@ -805,6 +824,9 @@ -- myOtherFunction x y = 'joinBV' $ MT x y -- @ -- +-- The 'BV' pattern synonym abstracts over manual application of 'splitBV' +-- and 'joinBV' as a pattern. +-- -- This will work with all data types made with a single constructor, whose -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'. @@ -821,6 +843,9 @@ -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'. The type also must derive 'Generic'. -- +-- Note that 'BV' is a pattern synonym version where the deconstructor is +-- exactly a view into 'splitBV'. +-- -- @since 0.2.2.0 splitBV :: ( Generic (z f) @@ -836,7 +861,7 @@ splitBV = E.splitBV E.addFunc E.addFuncs E.zeroFunc E.zeroFuncs {-# INLINE splitBV #-} --- | Split out a 'BVar' of "higher-kinded data type", a la +-- | Assemble a 'BVar' of "higher-kinded data type", a la -- -- -- It lets you take a 'BVar' of every field of a value, and join them into @@ -848,6 +873,9 @@ -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'. -- +-- Note that 'BV' is a pattern synonym version where the constructor is +-- exactly 'joinBV'. +-- -- @since 0.2.2.0 joinBV :: ( Generic (z f) @@ -862,3 +890,31 @@ -> BVar s (z f) -- ^ 'BVar' of combined value joinBV = E.joinBV E.addFunc E.addFuncs E.zeroFunc E.zeroFuncs {-# INLINE joinBV #-} + +-- | Pattern synonym wrapping manual usage of 'splitBV' and 'joinBV'. It +-- is a pattern for a @'BVar' s (z f)@ containing a @z ('BVar' s)@ +-- +-- @since 0.2.3.0 +pattern BV + :: ( Generic (z f) + , Generic (z (BVar s)) + , E.BVGroup s as (Rep (z f)) (Rep (z (BVar s))) + , Backprop (Rep (z f) ()) + , Backprop (z f) + , Every Backprop as + , Known Length as + , Reifies s W + ) +#if MIN_VERSION_base(4,10,0) + => z (BVar s) -- ^ 'BVar's of fields + -> BVar s (z f) -- ^ 'BVar' of combined value +#else + => z (BVar s) + -> BVar s (z f) +#endif +pattern BV v <- (splitBV->v) + where + BV = joinBV +#if MIN_VERSION_base(4,10,0) +{-# COMPLETE BV #-} +#endif diff --git a/src/Numeric/Backprop/Explicit.hs b/src/Numeric/Backprop/Explicit.hs --- a/src/Numeric/Backprop/Explicit.hs +++ b/src/Numeric/Backprop/Explicit.hs @@ -165,35 +165,40 @@ {-# INLINE auto #-} -- | 'Numeric.Backprop.backpropWithN', but with explicit 'zero'. +-- +-- Note that argument order changed in v0.2.3. +-- +-- @since 0.2.0.0 backpropWithN :: Prod ZeroFunc as -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, Tuple as) -backpropWithN zfs f xs g = backpropN zfs (OF g) f xs + -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function +backpropWithN zfs f = second (. OF) . backpropN zfs f {-# INLINE backpropWithN #-} -- | 'Numeric.Backprop.backprop', but with explicit 'zero' and 'one'. +-- +-- Note that argument order changed in v0.2.3. backprop :: ZeroFunc a - -> OneFunc b -> (forall s. Reifies s W => BVar s a -> BVar s b) -> a - -> (b, a) -backprop zfa ofb f = second (getI . head') - . backpropN (zfa :< Ø) ofb (f . head') - . only_ + -> (b, OneFunc b -> a) +backprop zfa f = second ((getI . head') .) + . backpropN (zfa :< Ø) (f . head') + . only_ {-# INLINE backprop #-} -- | 'Numeric.Backprop.backpropWith', but with explicit 'zero'. +-- +-- Note that argument order changed in v0.2.3. backpropWith :: ZeroFunc a -> (forall s. Reifies s W => BVar s a -> BVar s b) -> a - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, a) -backpropWith zfa f x g = backprop zfa (OF g) f x + -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function +backpropWith zfa f = second (. OF) . backprop zfa f {-# INLINE backpropWith #-} -- | 'evalBP' but with no arguments. Useful when everything is just given @@ -221,7 +226,7 @@ -> (forall s. Reifies s W => BVar s a -> BVar s b) -> a -> a -gradBP zfa ofb f = snd . backprop zfa ofb f +gradBP zfa ofb f = ($ ofb) . snd . backprop zfa f {-# INLINE gradBP #-} -- | 'Numeric.Backprop.gradBP', Nbut with explicit 'zero' and 'one'. @@ -231,34 +236,38 @@ -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as -> Tuple as -gradBPN zfas ofb f = snd . backpropN zfas ofb f +gradBPN zfas ofb f = ($ ofb) . snd . backpropN zfas f {-# INLINE gradBPN #-} -- | 'Numeric.Backprop.backprop2', but with explicit 'zero' and 'one'. +-- +-- Note that argument order changed in v0.2.3. backprop2 :: ZeroFunc a -> ZeroFunc b - -> OneFunc c -> (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c) -> a -> b - -> (c, (a, b)) -backprop2 zfa zfb ofc f x y = second (\(dx ::< dy ::< Ø) -> (dx, dy)) $ - backpropN (zfa :< zfb :< Ø) ofc + -> (c, OneFunc c -> (a, b)) +backprop2 zfa zfb f x y = second ((\(dx ::< dy ::< Ø) -> (dx, dy)) .) $ + backpropN (zfa :< zfb :< Ø) (\(x' :< y' :< Ø) -> f x' y') (x ::< y ::< Ø) {-# INLINE backprop2 #-} -- | 'Numeric.Backprop.backpropWith2', but with explicit 'zero'. +-- +-- Note that argument order changed in v0.2.3. +-- +-- @since 0.2.0.0 backpropWith2 :: ZeroFunc a -> ZeroFunc b -> (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c) -> a -> b - -> (c -> c) -- ^ Gradient of final result with respect to output of function - -> (c, (a, b)) -backpropWith2 zfa zfb f x y g = backprop2 zfa zfb (OF g) f x y + -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function +backpropWith2 zfa zfb f x = second (. OF) . backprop2 zfa zfb f x {-# INLINE backpropWith2 #-} -- | 'evalBP' for a two-argument function. See @@ -281,7 +290,7 @@ -> a -> b -> (a, b) -gradBP2 zfa zfb ofc f x = snd . backprop2 zfa zfb ofc f x +gradBP2 zfa zfb ofc f x = ($ ofc) . snd . backprop2 zfa zfb f x {-# INLINE gradBP2 #-} -- | 'Numeric.Backprop.isoVar' with explicit 'add' and 'zero'. diff --git a/src/Numeric/Backprop/Internal.hs b/src/Numeric/Backprop/Internal.hs --- a/src/Numeric/Backprop/Internal.hs +++ b/src/Numeric/Backprop/Internal.hs @@ -171,8 +171,8 @@ -- using 'Numeric.Backprop.liftOp' and related functions. This is how you -- can create primitive functions that users can use to manipulate your -- library's values. See --- --- for a detailed guide. +-- for a detailed +-- guide. -- -- For example, the /hmatrix/ library has a matrix-vector multiplication -- function, @#> :: L m n -> R n -> L m@. @@ -411,7 +411,7 @@ liftOp3 afa afb afc z o !v !u !w = unsafePerformIO $ liftOp3_ afa afb afc z o v u w {-# INLINE liftOp3 #-} --- TODO: can we get the zero and scale func from the bvar? +-- TODO: can we get the zero and add func from the bvar? viewVar_ :: forall a b s. Reifies s W => AddFunc a @@ -438,7 +438,7 @@ viewVar af z l !v = unsafePerformIO $ viewVar_ af z l v {-# INLINE viewVar #-} --- TODO: can zero and scale func be gotten from the input bvars? +-- TODO: can zero and add func be gotten from the input bvars? setVar_ :: forall a b s. Reifies s W => AddFunc a @@ -484,15 +484,14 @@ sequenceVar af z !v = unsafePerformIO $ traverseVar' af z id traverse v {-# INLINE sequenceVar #-} --- TODO: can scale funcs and zeros be had from bvars and Functor instance? +-- TODO: can add funcs and zeros be had from bvars and Functor instance? collectVar_ :: forall t a s. (Reifies s W, Foldable t, Functor t) => AddFunc a -> ZeroFunc a - -> ZeroFunc (t a) -> t (BVar s a) -> IO (BVar s (t a)) -collectVar_ af z z' !vs = withV (toList vs) $ \(vVec :: Vec n (BVar s a)) -> do +collectVar_ af z !vs = withV (toList vs) $ \(vVec :: Vec n (BVar s a)) -> do let tn :: TapeNode (t a) tn = TN { _tnInputs = vecToProd (vmap ((`IR` runAF af) . getI) vVec) @@ -501,18 +500,27 @@ . toList } traverse_ (evaluate . forceBVar) vs - insertNode tn (_bvVal <$> vs) z' (reflect (Proxy @s)) + insertNode tn (_bvVal <$> vs) (ZF $ fmap (runZF z)) (reflect (Proxy @s)) {-# INLINE collectVar_ #-} -- | 'Numeric.Backprop.collectVar', but with explicit 'add' and 'zero'. +-- +-- NOTE: Prior to v0.2.3, this required an extra @'ZeroFunc' (t a)@ input. +-- However, after v0.2.3, the 'ZeroFunc' is now derived from the 'Functor' +-- instance of @t@. This makes the API a little more convenient, and it +-- enforces consistency with the @'ZeroFunc' a@, so people can't pass in +-- nonsense combinations. +-- +-- Please submit an issue to the issue tracker if you find yourself in +-- a situation where you need the flexibility to provide a separte +-- @'ZeroFunc' a@ and @'ZeroFunc' (t a)@. collectVar :: forall t a s. (Reifies s W, Foldable t, Functor t) => AddFunc a -> ZeroFunc a - -> ZeroFunc (t a) -> t (BVar s a) -> BVar s (t a) -collectVar af z z' !vs = unsafePerformIO $ collectVar_ af z z' vs +collectVar af z !vs = unsafePerformIO $ collectVar_ af z vs {-# INLINE collectVar #-} traverseVar' @@ -613,18 +621,19 @@ {-# INLINE gradRunner #-} -- | 'Numeric.Backprop.backpropN', but with explicit 'zero' and 'one'. +-- +-- Note that argument order changed in v0.2.3. backpropN :: forall as b. () => Prod ZeroFunc as - -> OneFunc b -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as - -> (b, Tuple as) -backpropN zfs ofb f !xs = (y, g) + -> (b, OneFunc b -> Tuple as) +backpropN zfs f !xs = (y, g) where !(!tp@(!_,!_),!y) = unsafePerformIO $ fillWengert f xs - g :: Tuple as - g = runST $ do + g :: OneFunc b -> Tuple as + g ofb = runST $ do r <- initRunner tp $ bimap getSum (`appEndo` []) . fst $ zipWithPM_ go zfs xs diff --git a/src/Numeric/Backprop/Num.hs b/src/Numeric/Backprop/Num.hs --- a/src/Numeric/Backprop/Num.hs +++ b/src/Numeric/Backprop/Num.hs @@ -88,6 +88,7 @@ , Reifies ) where +import Data.Bifunctor import Data.Maybe import Data.Reflection import Data.Type.Index @@ -114,19 +115,22 @@ => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as -> (b, Tuple as) -backpropN = E.backpropN E.zfNums E.ofNum +backpropN f = second ($ E.ofNum) . E.backpropN E.zfNums f {-# INLINE backpropN #-} -- | 'Numeric.Backprop.backpropWithN', but with 'Num' constraints instead -- of 'Backprop' constraints. -- -- See 'backpropN' for information on the 'Every' constraint. +-- +-- Note that argument order changed in v0.2.3. +-- +-- @since 0.2.0.0 backpropWithN :: (Every Num as, Known Length as) => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b) -> Tuple as - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, Tuple as) + -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function backpropWithN = E.backpropWithN E.zfNums {-# INLINE backpropWithN #-} @@ -140,7 +144,7 @@ => (forall s. Reifies s W => BVar s a -> BVar s b) -> a -> (b, a) -backprop = E.backprop E.zfNum E.ofNum +backprop f = second ($ E.ofNum) . E.backprop E.zfNum f {-# INLINE backprop #-} -- | 'Numeric.Backprop.backpropWith', but with 'Num' constraints instead of @@ -148,12 +152,15 @@ -- -- See module documentation for "Numeric.Backprop.Num" for information on -- using this with tuples. +-- +-- Note that argument order changed in v0.2.3. +-- +-- @since 0.2.0.0 backpropWith :: Num a => (forall s. Reifies s W => BVar s a -> BVar s b) -> a - -> (b -> b) -- ^ Gradient of final result with respect to output of function - -> (b, a) + -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function backpropWith = E.backpropWith E.zfNum {-# INLINE backpropWith #-} @@ -185,18 +192,21 @@ -> a -> b -> (c, (a, b)) -backprop2 = E.backprop2 E.zfNum E.zfNum E.ofNum +backprop2 f x = second ($ E.ofNum) . E.backprop2 E.zfNum E.zfNum f x {-# INLINE backprop2 #-} -- | 'Numeric.Backprop.backpropWith2', but with 'Num' constraints instead of -- 'Backprop' constraints. +-- +-- Note that argument order changed in v0.2.3. +-- +-- @since 0.2.0.0 backpropWith2 :: (Num a, Num b) => (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c) -> a -> b - -> (c -> c) -- ^ Gradient of final result with respect to output of function - -> (c, (a, b)) + -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function backpropWith2 = E.backpropWith2 E.zfNum E.zfNum {-# INLINE backpropWith2 #-} @@ -342,14 +352,12 @@ -- | 'Numeric.Backprop.collectVar', but with 'Num' constraints instead of -- 'Backprop' constraints. -- --- If you are using a list or vector, I recommend using --- instead: --- it's a fixed-length vector type with a very appropriate 'Num' instance! +-- Prior to v0.2.3, required a 'Num' constraint on @t a@. collectVar - :: (Foldable t, Functor t, Num a, Num (t a), Reifies s W) + :: (Foldable t, Functor t, Num a, Reifies s W) => t (BVar s a) -> BVar s (t a) -collectVar = E.collectVar E.afNum E.zfNum E.zfNum +collectVar = E.collectVar E.afNum E.zfNum {-# INLINE collectVar #-} -- | 'Numeric.Backprop.liftOp', but with 'Num' constraints instead of diff --git a/src/Numeric/Backprop/Op.hs b/src/Numeric/Backprop/Op.hs --- a/src/Numeric/Backprop/Op.hs +++ b/src/Numeric/Backprop/Op.hs @@ -36,9 +36,8 @@ -- with 'BVar's using 'liftOp', 'liftOp1', 'liftOp2', and 'liftOp3'. -- -- If you are writing a library, see --- --- for a guide for equipping your library with backpropatable operations --- using 'Op's. +-- for a guide for +-- equipping your library with backpropatable operations using 'Op's. -- module Numeric.Backprop.Op ( diff --git a/src/Prelude/Backprop.hs b/src/Prelude/Backprop.hs --- a/src/Prelude/Backprop.hs +++ b/src/Prelude/Backprop.hs @@ -33,15 +33,19 @@ , toList , mapAccumL , mapAccumR + , foldr, foldl' -- * Functor and Applicative , fmap , (<$>) , pure , liftA2 , liftA3 - -- * Misc + -- * Numeric , fromIntegral , realToFrac + , round + , fromIntegral' + -- * Misc , E.coerce ) where @@ -100,19 +104,47 @@ maximum = E.maximum E.addFunc E.zeroFunc {-# INLINE maximum #-} +-- | Lifed 'P.foldr'. Essentially just 'toList' composed with a normal +-- list 'P.foldr', and is only here for convenience. +-- +-- @since 0.2.3.0 +foldr + :: (Traversable t, Backprop a, Reifies s W) + => (BVar s a -> BVar s b -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldr = E.foldr E.addFunc E.zeroFunc +{-# INLINE foldr #-} + +-- | Lifed 'P.foldl''. Essentially just 'toList' composed with a normal +-- list 'P.foldl'', and is only here for convenience. +-- +-- @since 0.2.3.0 +foldl' + :: (Traversable t, Backprop a, Reifies s W) + => (BVar s b -> BVar s a -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldl' = E.foldl' E.addFunc E.zeroFunc +{-# INLINE foldl' #-} + -- | Lifted 'P.fmap'. Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Functor's. +-- +-- Prior to v0.2.3, required a 'Backprop' constraint on @f b@. fmap - :: (Traversable f, Backprop a, Backprop b, Backprop (f b), Reifies s W) + :: (Traversable f, Backprop a, Backprop b, Reifies s W) => (BVar s a -> BVar s b) -> BVar s (f a) -> BVar s (f b) -fmap = E.fmap E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc +fmap = E.fmap E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE fmap #-} -- | Alias for 'fmap'. (<$>) - :: (Traversable f, Backprop a, Backprop b, Backprop (f b), Reifies s W) + :: (Traversable f, Backprop a, Backprop b, Reifies s W) => (BVar s a -> BVar s b) -> BVar s (f a) -> BVar s (f b) @@ -121,20 +153,24 @@ -- | Lifted 'P.traverse'. Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Functor's. +-- +-- Prior to v0.2.3, required a 'Backprop' constraint on @f (t b)@. traverse - :: (Traversable t, Applicative f, Foldable f, Backprop a, Backprop b, Backprop (f (t b)), Backprop (t b), Reifies s W) + :: (Traversable t, Applicative f, Foldable f, Backprop a, Backprop b, Backprop (t b), Reifies s W) => (BVar s a -> f (BVar s b)) -> BVar s (t a) -> BVar s (f (t b)) traverse = E.traverse E.addFunc E.addFunc E.addFunc - E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc + E.zeroFunc E.zeroFunc {-# INLINE traverse #-} -- | Lifted 'P.liftA2'. Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Applicative's. +-- +-- Prior to v0.2.3, required a 'Backprop' constraint on @f c@. liftA2 :: ( Traversable f, Applicative f - , Backprop a, Backprop b, Backprop c, Backprop (f c) + , Backprop a, Backprop b, Backprop c , Reifies s W ) => (BVar s a -> BVar s b -> BVar s c) @@ -142,15 +178,17 @@ -> BVar s (f b) -> BVar s (f c) liftA2 = E.liftA2 E.addFunc E.addFunc E.addFunc - E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc + E.zeroFunc E.zeroFunc E.zeroFunc {-# INLINE liftA2 #-} -- | Lifted 'P.liftA3'. Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Applicative's. +-- +-- Prior to v0.2.3, required a 'Backprop' constraint on @f d@. liftA3 :: ( Traversable f , Applicative f - , Backprop a, Backprop b, Backprop c, Backprop d, Backprop (f d) + , Backprop a, Backprop b, Backprop c, Backprop d , Reifies s W ) => (BVar s a -> BVar s b -> BVar s c -> BVar s d) @@ -159,7 +197,7 @@ -> BVar s (f c) -> BVar s (f d) liftA3 = E.liftA3 E.addFunc E.addFunc E.addFunc E.addFunc - E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc + E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc {-# INLINE liftA3 #-} -- | Lifted conversion between two 'P.Integral' instances. @@ -182,6 +220,35 @@ realToFrac = E.realToFrac E.addFunc E.zeroFunc {-# INLINE realToFrac #-} +-- | Lifted version of 'P.round'. +-- +-- Gradient should technically diverge whenever the fractional part is 0.5, +-- but does not do this for convenience reasons. +-- +-- @since 0.2.3.0 +round + :: (P.RealFrac a, P.Integral b, Reifies s W) + => BVar s a + -> BVar s b +round = E.round E.afNum E.zfNum +{-# INLINE round #-} + +-- | Lifted version of 'P.fromIntegral', defined to let you return +-- 'P.RealFrac' instances as targets, instead of only other 'P.Integral's. +-- Essentially the opposite of 'round'. +-- +-- The gradient should technically diverge whenever the fractional part of +-- the downstream gradient is 0.5, but does not do this for convenience +-- reasons. +-- +-- @since 0.2.3.0 +fromIntegral' + :: (P.Integral a, P.RealFrac b, Reifies s W) + => BVar s a + -> BVar s b +fromIntegral' = E.fromIntegral' E.afNum E.zfNum +{-# INLINE fromIntegral' #-} + -- | Lifted version of 'P.toList'. Takes a 'BVar' of a 'Traversable' of -- items and returns a list of 'BVar's for each item. -- @@ -200,25 +267,29 @@ -- | Lifted version of 'P.mapAccumL'. -- +-- Prior to v0.2.3, required a 'Backprop' constraint on @t b@. +-- -- @since 0.2.2.0 mapAccumL - :: (Traversable t, Backprop b, Backprop c, Backprop (t c), Reifies s W) + :: (Traversable t, Backprop b, Backprop c, Reifies s W) => (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumL = E.mapAccumL E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc +mapAccumL = E.mapAccumL E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE mapAccumL #-} -- | Lifted version of 'P.mapAccumR'. -- +-- Prior to v0.2.3, required a 'Backprop' constraint on @t b@. +-- -- @since 0.2.2.0 mapAccumR - :: (Traversable t, Backprop b, Backprop c, Backprop (t c), Reifies s W) + :: (Traversable t, Backprop b, Backprop c, Reifies s W) => (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumR = E.mapAccumR E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc +mapAccumR = E.mapAccumR E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE mapAccumR #-} diff --git a/src/Prelude/Backprop/Explicit.hs b/src/Prelude/Backprop/Explicit.hs --- a/src/Prelude/Backprop/Explicit.hs +++ b/src/Prelude/Backprop/Explicit.hs @@ -27,14 +27,18 @@ , toList , mapAccumL , mapAccumR + , foldr, foldl' -- * Functor and Applicative , fmap , pure , liftA2 , liftA3 - -- * Misc + -- * Numeric , fromIntegral , realToFrac + , round + , fromIntegral' + -- * Misc , coerce ) where @@ -129,21 +133,56 @@ ) {-# INLINE maximum #-} +-- | 'Prelude.Backprop.foldr', but taking explicit 'add' and 'zero'. +-- +-- @since 0.2.3.0 +foldr + :: (Traversable t, Reifies s W) + => AddFunc a + -> ZeroFunc a + -> (BVar s a -> BVar s b -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldr af zf f x = P.foldr f x . toList af zf +{-# INLINE foldr #-} + +-- | 'Prelude.Backprop.foldl'', but taking explicit 'add' and 'zero'. +-- +-- @since 0.2.3.0 +foldl' + :: (Traversable t, Reifies s W) + => AddFunc a + -> ZeroFunc a + -> (BVar s b -> BVar s a -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldl' af zf f x = P.foldl' f x . toList af zf +{-# INLINE foldl' #-} + -- | 'Prelude.Backprop.fmap', but taking explicit 'add' and 'zero'. +-- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f b)@ +-- parameter. fmap :: (Traversable f, Reifies s W) => AddFunc a -> AddFunc b -> ZeroFunc a -> ZeroFunc b - -> ZeroFunc (f b) -> (BVar s a -> BVar s b) -> BVar s (f a) -> BVar s (f b) -fmap afa afb zfa zfb zfbs f = collectVar afb zfb zfbs . P.fmap f . sequenceVar afa zfa +fmap afa afb zfa zfb f = collectVar afb zfb . P.fmap f . sequenceVar afa zfa {-# INLINE fmap #-} -- | 'Prelude.Backprop.traverse', but taking explicit 'add' and 'zero'. +-- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t b)@ +-- and @'ZeroFunc' (f (t b))@ parameters. traverse :: (Traversable t, Applicative f, Foldable f, Reifies s W) => AddFunc a @@ -151,19 +190,24 @@ -> AddFunc (t b) -> ZeroFunc a -> ZeroFunc b - -> ZeroFunc (t b) - -> ZeroFunc (f (t b)) -> (BVar s a -> f (BVar s b)) -> BVar s (t a) -> BVar s (f (t b)) -traverse afa afb aftb zfa zfb zftb zfftb f - = collectVar aftb zftb zfftb - . P.fmap (collectVar afb zfb zftb) +traverse afa afb aftb zfa zfb f + = collectVar aftb zftb + . P.fmap (collectVar afb zfb) . P.traverse f . sequenceVar afa zfa + where + zftb = ZF $ P.fmap (runZF zfb) + {-# INLINE zftb #-} {-# INLINE traverse #-} -- | 'Prelude.Backprop.liftA2', but taking explicit 'add' and 'zero'. +-- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f c)@ +-- parameter. liftA2 :: ( Traversable f , Applicative f @@ -175,18 +219,21 @@ -> ZeroFunc a -> ZeroFunc b -> ZeroFunc c - -> ZeroFunc (f c) -> (BVar s a -> BVar s b -> BVar s c) -> BVar s (f a) -> BVar s (f b) -> BVar s (f c) -liftA2 afa afb afc zfa zfb zfc zffc f x y - = collectVar afc zfc zffc +liftA2 afa afb afc zfa zfb zfc f x y + = collectVar afc zfc $ f P.<$> sequenceVar afa zfa x P.<*> sequenceVar afb zfb y {-# INLINE liftA2 #-} -- | 'Prelude.Backprop.liftA3', but taking explicit 'add' and 'zero'. +-- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f d)@ +-- parameter. liftA3 :: ( Traversable f , Applicative f @@ -200,14 +247,13 @@ -> ZeroFunc b -> ZeroFunc c -> ZeroFunc d - -> ZeroFunc (f d) -> (BVar s a -> BVar s b -> BVar s c -> BVar s d) -> BVar s (f a) -> BVar s (f b) -> BVar s (f c) -> BVar s (f d) -liftA3 afa afb afc afd zfa zfb zfc zfd zffd f x y z - = collectVar afd zfd zffd +liftA3 afa afb afc afd zfa zfb zfc zfd f x y z + = collectVar afd zfd $ f P.<$> sequenceVar afa zfa x P.<*> sequenceVar afb zfb y P.<*> sequenceVar afc zfc z @@ -227,8 +273,7 @@ -> ZeroFunc b -> BVar s a -> BVar s b -fromIntegral af zf = liftOp1 af zf . op1 $ \x -> - (P.fromIntegral x, P.fromIntegral) +fromIntegral af zf = isoVar af zf P.fromIntegral P.fromIntegral {-# INLINE fromIntegral #-} -- | 'Prelude.Backprop.realToFrac', but taking explicit 'add' and 'zero'. @@ -240,10 +285,34 @@ -> ZeroFunc b -> BVar s a -> BVar s b -realToFrac af zf = liftOp1 af zf . op1 $ \x -> - (P.realToFrac x, P.realToFrac) +realToFrac af zf = isoVar af zf P.realToFrac P.realToFrac {-# INLINE realToFrac #-} +-- | 'Prelude.Backprop.round', but taking explicit 'add' and 'zero'. +-- +-- @since 0.2.3.0 +round + :: (P.RealFrac a, P.Integral b, Reifies s W) + => AddFunc a + -> ZeroFunc b + -> BVar s a + -> BVar s b +round af zf = isoVar af zf P.round P.fromIntegral +{-# INLINE round #-} + +-- | 'Prelude.Backprop.fromIntegral'', but taking explicit 'add' and +-- 'zero'. +-- +-- @since 0.2.3.0 +fromIntegral' + :: (P.Integral a, P.RealFrac b, Reifies s W) + => AddFunc a + -> ZeroFunc b + -> BVar s a + -> BVar s b +fromIntegral' af zf = isoVar af zf P.fromIntegral P.round +{-# INLINE fromIntegral' #-} + -- | 'Prelude.Backprop.length', but taking explicit 'add' and 'zero'. -- -- @since 0.2.2.0 @@ -258,6 +327,10 @@ -- | 'Prelude.Backprop.mapAccumL', but taking explicit 'add' and 'zero'. -- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t c)@ +-- parameter. +-- -- @since 0.2.2.0 mapAccumL :: (Traversable t, Reifies s W) @@ -265,19 +338,22 @@ -> AddFunc c -> ZeroFunc b -> ZeroFunc c - -> ZeroFunc (t c) -> (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumL afb afc zfb zfc zftc f s = - second (collectVar afc zfc zftc) +mapAccumL afb afc zfb zfc f s = + second (collectVar afc zfc) . P.mapAccumL f s . sequenceVar afb zfb {-# INLINE mapAccumL #-} -- | 'Prelude.Backprop.mapAccumR', but taking explicit 'add' and 'zero'. -- +-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for +-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t c)@ +-- parameter. +-- -- @since 0.2.2.0 mapAccumR :: (Traversable t, Reifies s W) @@ -285,13 +361,12 @@ -> AddFunc c -> ZeroFunc b -> ZeroFunc c - -> ZeroFunc (t c) -> (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumR afb afc zfb zfc zftc f s = - second (collectVar afc zfc zftc) +mapAccumR afb afc zfb zfc f s = + second (collectVar afc zfc) . P.mapAccumR f s . sequenceVar afb zfb {-# INLINE mapAccumR #-} diff --git a/src/Prelude/Backprop/Num.hs b/src/Prelude/Backprop/Num.hs --- a/src/Prelude/Backprop/Num.hs +++ b/src/Prelude/Backprop/Num.hs @@ -26,15 +26,19 @@ , toList , mapAccumL , mapAccumR + , foldr, foldl' -- * Functor and Applicative , fmap , (<$>) , pure , liftA2 , liftA3 - -- * Misc + -- * Numeric , fromIntegral , realToFrac + , round + , fromIntegral' + -- * Misc , E.coerce ) where @@ -97,19 +101,47 @@ maximum = E.maximum E.afNum E.zfNum {-# INLINE maximum #-} +-- | 'Prelude.Backprop.foldr', but with 'Num' constraints instead of +-- 'Backprop' constraints. +-- +-- @since 0.2.3.0 +foldr + :: (Traversable t, Num a, Reifies s W) + => (BVar s a -> BVar s b -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldr = E.foldr E.afNum E.zfNum +{-# INLINE foldr #-} + +-- | 'Prelude.Backprop.foldl'', but with 'Num' constraints instead of +-- 'Backprop' constraints. +-- +-- @since 0.2.3.0 +foldl' + :: (Traversable t, Num a, Reifies s W) + => (BVar s b -> BVar s a -> BVar s b) + -> BVar s b + -> BVar s (t a) + -> BVar s b +foldl' = E.foldl' E.afNum E.zfNum +{-# INLINE foldl' #-} + -- | 'Prelude.Backprop.fmap', but with 'Num' constraints instead of -- 'Backprop' constraints. +-- +-- Prior to v0.2.3, required a 'Num' constraint on @f b@. fmap - :: (Traversable f, Num a, Num b, Num (f b), Reifies s W) + :: (Traversable f, Num a, Num b, Reifies s W) => (BVar s a -> BVar s b) -> BVar s (f a) -> BVar s (f b) -fmap = E.fmap E.afNum E.afNum E.zfNum E.zfNum E.zfNum +fmap = E.fmap E.afNum E.afNum E.zfNum E.zfNum {-# INLINE fmap #-} -- | Alias for 'fmap'. (<$>) - :: (Traversable f, Num a, Num b, Num (f b), Reifies s W) + :: (Traversable f, Num a, Num b, Reifies s W) => (BVar s a -> BVar s b) -> BVar s (f a) -> BVar s (f b) @@ -118,35 +150,44 @@ -- | 'Prelude.Backprop.traverse', but with 'Num' constraints instead of -- 'Backprop' constraints. +-- +-- See for +-- a fixed-length vector type with a very appropriate 'Num' instance! +-- +-- Prior to v0.2.3, required a 'Num' constraint on @f (t b)@. traverse - :: (Traversable t, Applicative f, Foldable f, Num a, Num b, Num (f (t b)), Num (t b), Reifies s W) + :: (Traversable t, Applicative f, Foldable f, Num a, Num b, Num (t b), Reifies s W) => (BVar s a -> f (BVar s b)) -> BVar s (t a) -> BVar s (f (t b)) -traverse = E.traverse E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum E.zfNum +traverse = E.traverse E.afNum E.afNum E.afNum E.zfNum E.zfNum {-# INLINE traverse #-} -- | 'Prelude.Backprop.liftA2', but with 'Num' constraints instead of -- 'Backprop' constraints. +-- +-- Prior to v0.2.3, required a 'Num' constraint on @f c@. liftA2 :: ( Traversable f , Applicative f - , Num a, Num b, Num c, Num (f c) + , Num a, Num b, Num c , Reifies s W ) => (BVar s a -> BVar s b -> BVar s c) -> BVar s (f a) -> BVar s (f b) -> BVar s (f c) -liftA2 = E.liftA2 E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum E.zfNum +liftA2 = E.liftA2 E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum {-# INLINE liftA2 #-} -- | 'Prelude.Backprop.liftA3', but with 'Num' constraints instead of -- 'Backprop' constraints. +-- +-- Prior to v0.2.3, required a 'Num' constraint on @f d@. liftA3 :: ( Traversable f , Applicative f - , Num a, Num b, Num c, Num d, Num (f d) + , Num a, Num b, Num c, Num d , Reifies s W ) => (BVar s a -> BVar s b -> BVar s c -> BVar s d) @@ -155,7 +196,7 @@ -> BVar s (f c) -> BVar s (f d) liftA3 = E.liftA3 E.afNum E.afNum E.afNum E.afNum - E.zfNum E.zfNum E.zfNum E.zfNum E.zfNum + E.zfNum E.zfNum E.zfNum E.zfNum {-# INLINE liftA3 #-} -- | 'Prelude.Backprop.fromIntegral', but with 'Num' constraints instead of @@ -180,6 +221,28 @@ realToFrac = E.realToFrac E.afNum E.zfNum {-# INLINE realToFrac #-} +-- | 'Prelude.Backprop.round', but with 'Num' constraints instead of +-- 'Backprop' constraints. +-- +-- @since 0.2.3.0 +round + :: (P.RealFrac a, P.Integral b, Reifies s W) + => BVar s a + -> BVar s b +round = E.round E.afNum E.zfNum +{-# INLINE round #-} + +-- | 'Prelude.Backprop.fromIntegral'', but with 'Num' constraints instead +-- of 'Backprop' constraints. +-- +-- @since 0.2.3.0 +fromIntegral' + :: (P.Integral a, P.RealFrac b, Reifies s W) + => BVar s a + -> BVar s b +fromIntegral' = E.fromIntegral' E.afNum E.zfNum +{-# INLINE fromIntegral' #-} + -- | 'Prelude.Backprop.toList', but with 'Num' constraints instead of -- 'Backprop' constraints. -- @@ -194,26 +257,30 @@ -- | 'Prelude.Backprop.mapAccumL', but with 'Num' constraints instead of -- 'Backprop' constraints. -- +-- Prior to v0.2.3, required a 'Num' constraint on @t b@. +-- -- @since 0.2.2.0 mapAccumL - :: (Traversable t, Num b, Num c, Num (t c), Reifies s W) + :: (Traversable t, Num b, Num c, Reifies s W) => (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumL = E.mapAccumL E.afNum E.afNum E.zfNum E.zfNum E.zfNum +mapAccumL = E.mapAccumL E.afNum E.afNum E.zfNum E.zfNum {-# INLINE mapAccumL #-} -- | 'Prelude.Backprop.mapAccumR', but with 'Num' constraints instead of -- 'Backprop' constraints. -- +-- Prior to v0.2.3, required a 'Num' constraint on @t b@. +-- -- @since 0.2.2.0 mapAccumR - :: (Traversable t, Num b, Num c, Num (t c), Reifies s W) + :: (Traversable t, Num b, Num c, Reifies s W) => (BVar s a -> BVar s b -> (BVar s a, BVar s c)) -> BVar s a -> BVar s (t b) -> (BVar s a, BVar s (t c)) -mapAccumR = E.mapAccumR E.afNum E.afNum E.zfNum E.zfNum E.zfNum +mapAccumR = E.mapAccumR E.afNum E.afNum E.zfNum E.zfNum {-# INLINE mapAccumR #-}