backprop 0.2.2.0 → 0.2.3.0

raw patch · 18 files changed

+2328/−230 lines, 18 filesPVP: major bump suggested

API removals or changes: PVP suggests a major version bump

API changes (from Hackage documentation)

Files

• CHANGELOG.md +24/−0
• README.md +76/−104
• backprop.cabal +12/−5
• doc/01-getting-started.md +227/−0
• doc/02-a-detailed-look.md +127/−0
• doc/03-manipulating-bvars.md +430/−0
• doc/04-the-backprop-typeclass.md +132/−0
• doc/05-applications.md +48/−0
• doc/06-equipping-your-library.md +776/−0
• doc/index.md +61/−0
• src/Numeric/Backprop.hs +72/−16
• src/Numeric/Backprop/Explicit.hs +30/−21
• src/Numeric/Backprop/Internal.hs +24/−15
• src/Numeric/Backprop/Num.hs +22/−14
• src/Numeric/Backprop/Op.hs +2/−3
• src/Prelude/Backprop.hs +85/−14
• src/Prelude/Backprop/Explicit.hs +99/−24
• src/Prelude/Backprop/Num.hs +81/−14

@@ -1,6 +1,30 @@ Changelog ========= +Version 0.2.3.0+---------------++*May 25, 2018*++<https://github.com/mstksg/backprop/releases/tag/v0.2.3.0>++*   Argument order in `backpropWith` family of functions switched around to+    allow for final gradient to be given after-the-fact.  **Braking change**+    for anyone using any `backpropWith` function.+*   As a consequence of the previous change, `backprop` family of functions in+    *Explicit* interfaces also all changed argument order.  **Breaking change**+    only for those using the *Explicit* interfaces.+*   Explicit `collectVar` no longer needs a `ZeroFunc` for the container, and+    so all versions of `collectVar` and functions that use it (`fmap`,+    `liftA2`, `liftA3`, `traverse`, `mapAccumL`, `mapAccumR`) no longer require+    `Backprop` or `Num` instances for the final returned container type.  This+    enables a lot more flexibility in container types.  **Breaking change**+    only for those using the *Explicit* interfaces.+*   `BV` pattern synonym added to *Numeric.Backprop*, abstracting over+    application of `splitBV` and `joinBV`.+*   `foldr` and `foldl'` added to Prelude modules, for convenience.+*   `round` and `fromIntegral'` ("unround") added to Prelude modules.+ Version 0.2.2.0 --------------- 

@@ -1,37 +1,54 @@-backprop-========--[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)+[backprop][docs]+================ -[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=2592000)](https://hackage.haskell.org/package/backprop)+[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=86400)](https://hackage.haskell.org/package/backprop)+[![backprop on Stackage LTS 11](http://stackage.org/package/backprop/badge/lts-11)](http://stackage.org/lts-11/package/backprop)+[![backprop on Stackage Nightly](http://stackage.org/package/backprop/badge/nightly)](http://stackage.org/nightly/package/backprop) [![Build Status](https://travis-ci.org/mstksg/backprop.svg?branch=master)](https://travis-ci.org/mstksg/backprop) -[**Introductory blog post**][blog]+[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)+[![Beerpay](https://beerpay.io/mstksg/backprop/badge.svg?style=beer-square)](https://beerpay.io/mstksg/backprop) -[blog]: https://blog.jle.im/entry/introducing-the-backprop-library.html+[**Documentation and Walkthrough**][docs] +[docs]: https://backprop.jle.im+ Automatic *heterogeneous* back-propagation.  Write your functions to compute your result, and the library will automatically generate functions to compute your gradient.  Differs from [ad][] by offering full heterogeneity -- each intermediate step-and the resulting value can have different types.  Mostly intended for usage-with gradient descent and other numeric optimization techniques.+and the resulting value can have different types (matrices, vectors, scalars,+lists, etc.).  [ad]: http://hackage.haskell.org/package/ad -Currently up on [hackage][] (with 100% documentation coverage), but more-up-to-date documentation is currently rendered [on github pages][docs]!+Useful for applications in [differential programming][dp] and deep learning for+creating and training numerical models, especially as described in this blog+post on [a purely functional typed approach to trainable models][models].+Overall, intended for the implementation of gradient descent and other numeric+optimization techniques.  Comparable to the python library [autograd][]. +[dp]: https://www.facebook.com/yann.lecun/posts/10155003011462143+[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html+[autograd]: https://github.com/HIPS/autograd++Currently up on [hackage][], with haddock documentation!  However, a proper+library introduction and usage tutorial [is available here][docs].  See also my+[introductory blog post][blog].  You can also find help or support on the+[gitter channel][gitter].+ [hackage]: http://hackage.haskell.org/package/backprop-[docs]: https://mstksg.github.io/backprop+[blog]: https://blog.jle.im/entry/introducing-the-backprop-library.html+[gitter]: https://gitter.im/haskell-backprop/Lobby  If you want to provide *backprop* for users of your library, see this **[guide to equipping your library with backprop][library]**. -[library]: https://github.com/mstksg/backprop/wiki/Equipping-your-Library-with-Backprop+[library]: https://backprop.jle.im/06-equipping-your-library.html + MNIST Digit Classifier Example ------------------------------ @@ -61,7 +78,8 @@ Brief example ------------- -(This is a really brief version of my [blog post][blog])+(This is a really brief version of [the documentation walkthrough][docs] and my+[blog post][blog])  The quick example below describes the running of a neural network with one hidden layer to calculate its squared error with respect to target `targ`,@@ -74,138 +92,88 @@ [lens]: http://hackage.haskell.org/package/lens  ```haskell-data Network i h o = Net { _weight1 :: L h i-                         , _bias1   :: R h-                         , _weight2 :: L o h-                         , _bias2   :: R o-                         }+import Numeric.LinearAlgebra.Static.Backprop +data Network = Net { _weight1 :: L 20 100+                   , _bias1   :: R 20+                   , _weight2 :: L  5  20+                   , _bias2   :: R  5+                   }+ makeLenses ''Network ``` -Normally, we might write code to "run" a neural network on an input like this:+(`R n` is an n-length vector, `L m n` is an m-by-n matrix, etc., `#>` is+matrix-vector multiplication) +"Running" a network on an input vector might look like this:+ ```haskell-neuralNet-    :: R i-    -> Network i h o-    -> R h-neuralNet x n = z+runNet net x = z   where-    y = logistic $ (n ^. weight1) #> x + (n ^. bias1)-    z = logistic $ (n ^. weight2) #> y + (n ^. bias2)+    y = logistic $ (net ^^. weight1) #> x + (net ^^. bias1)+    z = logistic $ (net ^^. weight2) #> y + (net ^^. bias2)  logistic :: Floating a => a -> a logistic x = 1 / (1 + exp (-x)) ``` -(`R i` is an i-length vector, `L h i` is an h-by-i matrix, etc., `#>` is-matrix-vector multiplication, and `^.` is access to a field via lens.)--When given an input vector and the network, we compute the result of the neural-network ran on the input vector.+And that's it!  `neuralNet` is now backpropagatable! -We can write it, instead, using *backprop*:+We can "run" it using `evalBP`:  ```haskell-neuralNet-    :: Reifies s W-    => BVar s (R i)-    -> BVar s (Network i h o)-    -> BVar s (R o)-neuralNet x n = z-  where-    y = logistic $ (n ^^. weight1) #> x + (n ^^. bias1)-    z = logistic $ (n ^^. weight2) #> y + (n ^^. bias2)--logistic :: Floating a => a -> a-logistic x = 1 / (1 + exp (-x))+evalBP2 runNet :: Network -> R 100 -> R 5 ``` -(`#>!` is a backprop-aware version of `#>`, and `^^.` is access to a field via-lens in a `BVar`)+If we write a function to compute errors: -And that's it!  `neuralNet` is now backpropagatable!+```haskell+squaredError target output = error `dot` error+  where+    error = target - output+``` -We can "run" it using `evalBP`:+we can "test" our networks:  ```haskell-evalBP (neuralNet (constVar x)) :: Network i h o -> R o+netError target input net = squaredError (auto target)+                                         (runNet net (auto input)) ``` -And we can find the gradient using `gradBP`:+This can be run, again:  ```haskell-gradBP (neuralNet (constVar x)) :: Network i h o -> Network i h o+evalBP (netError myTarget myVector) :: Network -> Network ``` -If we write a function to compute errors:+Now, we just wrote a *normal function to compute the error of our network*.+With the *backprop* library, we now also have a way to *compute the gradient*,+as well!  ```haskell-netError-    :: Reifies s W-    => BVar s (R i)-    -> BVar s (R o)-    -> BVar s (Network i h o)-    -> BVar s Double-netError x targ n = norm_2 (neuralNet x - t)+gradBP (netError myTarget myVector) :: Network -> Network ``` -(`norm_2` is a backprop-aware euclidean norm)- Now, we can perform gradient descent!  ```haskell gradDescent-    :: R i-    -> R o-    -> Network i h o-    -> Network i h o+    :: R 100+    -> R 5+    -> Network+    -> Network gradDescent x targ n0 = n0 - 0.1 * gradient   where-    gradient = gradBP (netError (constVar x) (constVar targ)) n0+    gradient = gradBP (netError targ x) n0 ```  Ta dah!  We were able to compute the gradient of our error function, just by only saying how to compute *the error itself*. -For a more fleshed out example, see my [blog post][blog] and the [MNIST-tutorial][mnist-lhs] (also [rendered as a pdf][mnist-pdf])--Lens Access--------------A lot of the friction of dealing with `BVar s a`s instead of `a`s directly is-alleviated with the lens interface.--With a lens, you can "view" and "set" items inside a `BVar`, as if they were-the actual values:--```haskell-(^.)  ::        a -> Lens' a b ->        b-(^^.) :: BVar s a -> Lens' a b -> BVar s b--(.~)  :: Lens' a b ->        b ->        a ->        a-(.~~) :: Lens' a b -> BVar s b -> BVar s a -> BVar s a-```--And you can also extract multiple potential targets, as well, using-`Traversal`s and `Prism`s:--```haskell--- | Actually takes a Traversal, to be more general.--- Can be used to implement "pattern matching" on BVars-(^?)  ::        a -> Prism' a b -> Maybe (       b)-(^^?) :: BVar s a -> Prism' a b -> Maybe (BVar s b)--(^..)  ::        a -> Traversal' a b -> [       b]-(^^..) :: BVar s a -> Traversal' a b -> [BVar s b]-```--Note that the library itself has no *lens* dependency, using *[microlens][]*-instead.--[microlens]: http://hackage.haskell.org/package/microlens+For a more fleshed out example, see [the documentaiton][docs], my [blog+post][blog] and the [MNIST tutorial][mnist-lhs] (also [rendered as a+pdf][mnist-pdf])  Benchmarks ----------@@ -263,3 +231,7 @@      b.  How to support "monadic" operations that depend on results of previous         operations? (`ApBP` already exists for situations that don't)++    c.  What needs to be done to allow us to automatically do second,+        third-order differentiation, as well?  This might be useful for certain+        ODE solvers which rely on second order gradients and hessians.

@@ -2,20 +2,20 @@ -- -- see: https://github.com/sol/hpack ----- hash: d347cf6994856b821bb3cf3172a4b5ec8f0d39b680e29e39a019d89cf022b2a5+-- hash: 983ab7f63f7a2d0309dc97c50875836220f6cbd8535600ad8cbbef9fe0672195  name:           backprop-version:        0.2.2.0-synopsis:       Heterogeneous automatic differentation (backpropagation)+version:        0.2.3.0+synopsis:       Heterogeneous automatic differentation description:    Write your functions to compute your result, and the library will                 automatically generate functions to compute your gradient.                 .                 Implements heterogeneous reverse-mode automatic differentiation, commonly                 known as "backpropagation".                 .-                See <https://github.com/mstksg/backprop#readme README.md>+                See <https://backprop.jle.im> for official introduction and documentation. category:       Math-homepage:       https://github.com/mstksg/backprop#readme+homepage:       https://backprop.jle.im bug-reports:    https://github.com/mstksg/backprop/issues author:         Justin Le maintainer:     justin@jle.im@@ -28,6 +28,13 @@ extra-source-files:     Build.hs     CHANGELOG.md+    doc/01-getting-started.md+    doc/02-a-detailed-look.md+    doc/03-manipulating-bvars.md+    doc/04-the-backprop-typeclass.md+    doc/05-applications.md+    doc/06-equipping-your-library.md+    doc/index.md     README.md     renders/backprop-mnist.md     renders/backprop-mnist.pdf

@@ -0,0 +1,227 @@+---+title: Getting Started+---++Getting Started+===============++```haskell top hide+{-# LANGUAGE DataKinds        #-}+{-# LANGUAGE DeriveGeneric    #-}+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE TemplateHaskell  #-}+{-# LANGUAGE ViewPatterns     #-}+++import           GHC.Generics (Generic)+import           GHC.TypeNats+import           Inliterate.Import+import           Lens.Micro+import           Lens.Micro.TH+import           Numeric.Backprop.Class+import           Numeric.LinearAlgebra.Static (L, R)+import           System.Random+import qualified Numeric.LinearAlgebra.Static as H+```++*backprop* is a Haskell library available on hackage, so can be used in your+package however way you like to require libraries.  Be sure to add it to your+cabal file's (or package.yaml's) build-depends field.++Automatic Backpropagated Functions+----------------------------------++With *backprop*, you can write your functions in Haskell as normal functions:++```haskell top+import Numeric.Backprop++myFunc x = sqrt (x * 4)+```++They can be run with `evalBP`:++```haskell eval+evalBP myFunc (9 :: Double)+```++And...the twist?  You can also get the gradient of your functions!++```haskell eval+gradBP myFunc (9 :: Double)+```++And that's the gist of the entire library: write your functions to compute your+things, and `gradBP` will give you the gradients and derivatives of those+functions.++### Multiple Same-Type Inputs++Multiple inputs of the same type can be handled with `sequenceVar`:++```haskell top+funcOnList (sequenceVar->[x,y,z]) = sqrt (x / y) * z+```++```haskell eval+evalBP funcOnList [3,5,-2] :: Double+```++```haskell eval+gradBP funcOnList [3,5,-2] :: [Double]+```++Heterogeneous Backprop+----------------------++But the real magic happens when you mix and match types.  Let's make a simple+type representing a feed-forward fully connected artificial neural network with+100 inputs, a single hidden layer of 20 nodes, and 5 outputs:++```haskell top+data Net = N { _nWeights1 :: L 20 100+             , _nBias1    :: R 20+             , _nWeights2 :: L  5  20+             , _nBias2    :: R  5+             }+  deriving (Show, Generic)++instance Backprop Net++makeLenses ''Net+```++using the `L m n` type from the *[hmatrix][]* library to represent an m-by-n+matrix, and the `R n` type to represent an n-vector.++[hmatrix]: http://hackage.haskell.org/package/hmatrix++We can write a function to "run" the network on a `R 100` and get an `R 5`+back, using `^^.` for lens access and `#>` from the *[hmatrix-backprop][]* library for+matrix-vector multiplication:++[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop++```haskell top hide+instance Backprop (R n) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++instance (KnownNat n, KnownNat m) => Backprop (L n m) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++(#>)+    :: (KnownNat n, KnownNat m, Reifies s W)+    => BVar s (L n m) -> BVar s (R m) -> BVar s (R n)+(#>) = liftOp2 . op2 $ \xs y ->+    ( xs H.#> y+    , \d -> (d `H.outer` y, H.tr xs H.#> d)+    )++dot :: (KnownNat n, Reifies s W) => BVar s (R n) -> BVar s (R n) -> BVar s Double+dot = liftOp2 . op2 $ \x y ->+    ( x `H.dot` y+    , \d -> let d' = H.konst d+            in  (d' * y, x * d')+    )+```++```haskell top+runNet net x = z+  where+    -- run first layer+    y = logistic $ (net ^^. nWeights1) #> x + (net ^^. nBias1)+    -- run second layer+    z = logistic $ (net ^^. nWeights2) #> y + (net ^^. nBias2)++logistic :: Floating a => a -> a+logistic x = 1 / (1 + exp (-x))+```++We can *run* this with a network and input vector:++```haskell top hide+myVector :: R 100+myVector = H.randomVector 93752345 H.Uniform - 0.5++myTarget :: R 5+myTarget = H.randomVector 93752345 H.Uniform - 0.5++myNet :: Net+myNet = N (H.uniformSample 2394834 (-0.5) 0.5)+          (H.randomVector 84783451 H.Uniform - 0.5)+          (H.uniformSample 9293092 (-0.5) 0.5)+          (H.randomVector 64814524 H.Uniform - 0.5)++instance KnownNat n => AskInliterate (R n) where+    askInliterate = answerWith (show . H.extract)+instance AskInliterate Net where+    askInliterate = answerWith (unlines . (++ ["-- ..."]) . take 5 . lines . show)+```++```haskell eval+evalBP2 runNet myNet myVector+```++But --- and here's the fun part --- if we write a "loss function" to evaluate+"how badly" our network has done, using `dot` from the *hmatrix-backprop*+library:++```haskell top+squaredError target output = error `dot` error+  where+    error = target - output+```++we can "test" our networks:++```haskell top+netError target input net = squaredError (auto target)+                                         (runNet net (auto input))+```++(more on `auto` later)++```haskell eval+evalBP (netError myTarget myVector) myNet+```++At this point, we've *written a normal function to compute the error of our+network*.  And, with the backprop library...we now have a way to compute the+*gradient* of our network's error with respect to all of our weights!++```haskell eval+gradBP (netError myTarget myVector) myNet+```++We can now use the gradient to "[train][]" our network to give the correct+responses given a certain input!  This can be done by computing the gradient+for every expected input-output pair, and adjusting the network in the opposite+direction of the gradient every time.++[train]: https://blog.jle.im/entry/purely-functional-typed-models-1.html++Main Idea+---------++The main pattern of usage for this library is:++1.  Write your function normally to compute something (like the loss function)+2.  Use `gradBP` to automatically get the gradient of that something with+    respect to your inputs!++In the case of optimizing models, you:++1.  Write your function normally to compute the thing you want to minimize+2.  Use `gradBP` to automatically get the gradient of the thing you want to+    minimize with respect to your inputs.  Then, adjust your inputs according+    to this gradient until you get the perfect minimal result!++Now that you've had a taste, let's **[look at the details][details]**.  You can+also just go ahead and **[jump into the haddock documentation][haddock]**!++[details]: https://backprop.jle.im/02-a-detailed-look.html+[haddock]: https://hackage.haskell.org/package/backprop

@@ -0,0 +1,127 @@+---+title: A Detailed Look+---++A Detailed Look+===============++```haskell top hide+{-# LANGUAGE FlexibleContexts #-}++import           Numeric.Backprop+```++So, what's really going on?++The BVar+--------++The entire library revolves around the `BVar`, a variable holding a+"backpropagatable value".  As you use a `BVar`, the *backprop* library will+track how it is used and where you use it.  You can use `evalBP` to simply get+the result, but using `gradBP` will perform backpropagation ("reverse-mode+[automatic differentiation][autodiff]")++[autodiff]: https://en.wikipedia.org/wiki/Automatic_differentiation++For example, we looked earlier at a function that computes the square root of a+quadrupled number:++```haskell top+myFunc :: Double+       -> Double+myFunc x = sqrt (x * 4)+```++As we are using it, its type is "really":++```haskell top+myFunc' :: Reifies s W+        => BVar s Double+        -> BVar s Double+myFunc' x = sqrt (x * 4)+```++`myFunc'` takes a `BVar s Double` (a `BVar` containing a `Double`) and returns+a new one that is the square root of the quadrupled number.  You can think of+the `Reifies s W` as being a necessary constraint that allows backpropagation+to happen.++`BVar`s have `Num`, `Fractional`, and `Floating` instances, and so can be used+with addition, multiplication, square rooting, etc.  The "most general" type of+`myFunc` is `myFunc :: Floating a => a -> a`, and since `BVar s Double` has a+`Floating` instance, you could even just use it directly as a backpropagatable+function.++This means you can basically treat a `BVar s Double` almost exactly like it was+a `Double` --- you'll practically never tell the difference!  `BVar`s also have+`Ord` and `Eq` instances, so you can compare them and branch on the results,+too.++```haskell top+myAbs :: Reifies s W+      => BVar s Double+      -> BVar s Double+myAbs x | x < 0     = negate x+        | otherwise = x+```++The goal of the `BVar` interface is that you should be able to treat a `BVar s+a` (a `BVar` containing an `a`) as if it was an `a`, with no easily noticeable+differences.++Runners+-------++The entire point of the library is to write your computation as a normal+function taking a `BVar` (or many) and returning a single `BVar`.  Just treat+`BVar`s as if they actually were the value they are containing, and you can't+go wrong.++Once you do this, you can use `evalBP` to "run" the function itself:++```haskell+evalBP :: (forall s. Reifies s W => BVar s a -> BVar s b)+       -> (a -> b)+```++This can be read as taking a `BVar s a -> BVar s b` and returning the `a -> b`+that that function encodes.  The RankN type there (the `forall s.`) is mostly+there to prevent leakage of `BVar`s (same as it is used in *Control.Monad.ST*+and `runST`).  It ensures that no `BVar`s "escape" the function somehow.++`evalBP` is extremely efficient, and usually carries virtually zero overhead+over writing your function directly on your values without `BVar`s.++*But*, the more interesting thing of course is computing the *gradient* of your+function.  This is done with `gradBP`:++```haskell+gradBP :: (Backprop a, Backprop b)+       => (forall s. Reifies s W => BVar s a -> BVar s b)+       -> a+       -> a+```++Which takes a `BVar s a -> BVar s b` backpropagatable function and an input,+and returns *the gradient at that input*.  It gives the direction of greatest+positive change (in the output) of your input, and also how much a variation in+your input will affect your output.++And that's all there is to it!  Instead of `a -> b`'s, write `BVar s a -> BVar+s b`'s to compute what you want to know the gradient of.  These are normal+functions, so you can use all of your favorite higher order functions and+combinators (like `(.)`, `map`, etc.).  And once you're done, use `gradBP` to+compute that gradient.++Note that `gradBP` requires a `Backprop` constraint on the input and output of+your function.  `Backprop` is essentially the typeclass of values that can be+"backpropagated".  For product types, this instance is automatically derivable.+But writing your own custom instances for your own types is also fairly+straightforward.  More on this later!++The rest of the package really is just ways to manipulate `BVar s a`s as if+they were just `a`s, to make everything as smooth as possible.  Let's move on+to learning about **[ways to manipulate BVars][bvars]**!++[bvars]: https://backprop.jle.im/03-manipulating-bvars.html

@@ -0,0 +1,430 @@+---+title: Manipulating BVars+---++Manipulating BVars+==================++```haskell top hide+{-# LANGUAGE DataKinds          #-}+{-# LANGUAGE DeriveGeneric      #-}+{-# LANGUAGE FlexibleContexts   #-}+{-# LANGUAGE FlexibleInstances  #-}+{-# LANGUAGE StandaloneDeriving #-}+{-# LANGUAGE TemplateHaskell    #-}+{-# LANGUAGE TypeFamilies       #-}+{-# LANGUAGE ViewPatterns       #-}+++import           Data.Functor.Identity+import           GHC.Generics (Generic)+import           GHC.TypeNats+import           Inliterate.Import+import           Lens.Micro+import           Lens.Micro.TH+import           Numeric.Backprop+import           Numeric.Backprop.Class+import           Numeric.LinearAlgebra.Static (L, R)+import           System.Random+import qualified Numeric.LinearAlgebra.Static as H+```++The most important aspect of the usability of this library is allowing you to+seamlessly manipulate `BVar s a`s as if they were just `a`s, without requiring+you as the user to be able to recognize or acknowledge the difference.  Here+are some techniques to that end.++Remember, a `BVar s a` is a `BVar` containing an `a` --- it's an `a` that, when+used, keeps track of and propagates your gradient.++Typeclass Interface+-------------------++`BVar`s have `Num`, `Fractional`, `Floating`, `Eq`, and `Ord` instances.  These+instances are basically "lifted" to the `BVar` itself, so if you have a `BVar s+Double`, you can use `(*)`, `sqrt`, `(>)`, etc. on it exactly as if it were+just a `Double`.++Constant Values+---------------++If we don't *care* about a value's gradient, we can use `auto`:++```haskell+auto :: a -> BVar s a+```++`auto x` basically gives you a `BVar` that contains just `x` alone.  Useful for+using with functions that expect `BVar`s, but you just have a specific value+you want to use.++Coercible+---------++If `a` and `b` are `Coercible`, then so are `BVar s a` and `BVar s b`, using+the `coerceVar` function.  This is useful for "unwrapping" and "wrapping"+`BVar`s of newtypes:+++```haskell top+newtype MyInt = MyInt Int++getMyInt :: BVar s MyInt -> BVar s Int+getMyInt = coerceVar+```++Accessing Contents+------------------++The following techniques can be used to access values inside `BVar`s:++### Traversable Containers++One that we saw earlier was `sequenceVar`, which we used to turn a `BVar`+containing a list into a list of `BVar`s:++```haskell+sequenceVar :: (Backprop a, Reifies s W)+            => BVar s [a]+            -> [BVar s a]+```++If you have a `BVar` containing a list, you can get a list of `BVar`s of all of+that list's elements.  (`sequenceVar` actually works on all `Traversable`+instances, not just lists)  This is very useful when combined with+`-XViewPatterns`, as seen earlier.++### Records and Fields++In practice, a lot of usage involves functions involving contents of records or+data types containing fields.  The previous example, involving a simple ANN,+demonstrates this:++```haskell top+data Net = N { _nWeights1 :: L 20 100+             , _nBias1    :: R 20+             , _nWeights2 :: L  5  20+             , _nBias2    :: R  5+             }+  deriving (Show, Generic)++instance Backprop Net       -- can be automatically defined+```++To compute the result of this network (ran on an `R 100`, a 100-vector) and get+the output `R 5`, we need do a matrix multiplication by the `_nWeights1` field,+add the result to the `_nBias1` field...basically, the result is a function of+linear algebra and related operations on the input and all of the contents of+the `Net` data type.  However, you can't directly use `_nWeights`, since it+takes a `Net`, not `BVar s Net`.  And you also can't directly pattern match on+the `N` constructor.++There are two main options for this: the lens interface, and the higher-kinded+data interface.++#### Lens Interface++The most straightforward way to do this is the lens-based interface, using+`viewVar` or `^^.`.++If we make lenses for `Net` using the *[lens][]* or *[microlens-th][]* packages:++[lens]: http://hackage.haskell.org/package/lens+[microlens-th]: http://hackage.haskell.org/package/microlens-th++```haskell top+makeLenses ''Net+```++Then `^.` from the *lens* or *[microlens][]* packages lets you retrieve a field+from a `Net`:++[microlens]: http://hackage.haskell.org/package/microlens++```haskell+(^. nWeights1) :: Net -> L 20 100+(^. nBias1   ) :: Net -> R 20+(^. nWeights2) :: Net -> L  5  20+(^. nBias2   ) :: Net -> R  5+```++And, `^^.` from *backprop* (also aliased as `viewVar`) lets you do the same+thing from a `BVar s Net` (a `BVar` containing your `Net`):++```haskell+(^^. nWeights1) :: BVar s Net -> BVar s (L 20 100)+(^^. nBias1   ) :: BVar s Net -> BVar s (R 20)+(^^. nWeights2) :: BVar s Net -> BVar s (L  5  20)+(^^. nBias2   ) :: BVar s Net -> BVar s (R  5)+```++```haskell top hide+instance Backprop (R n) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++instance (KnownNat n, KnownNat m) => Backprop (L n m) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++(#>)+    :: (KnownNat n, KnownNat m, Reifies s W)+    => BVar s (L n m) -> BVar s (R m) -> BVar s (R n)+(#>) = liftOp2 . op2 $ \xs y ->+    ( xs H.#> y+    , \d -> (d `H.outer` y, H.tr xs H.#> d)+    )++dot :: (KnownNat n, Reifies s W) => BVar s (R n) -> BVar s (R n) -> BVar s Double+dot = liftOp2 . op2 $ \x y ->+    ( x `H.dot` y+    , \d -> let d' = H.konst d+            in  (d' * y, x * d')+    )+```++With our lenses and `^^.`, we can write our network running function.  This+time, I'll include the type!++```haskell top+runNet :: Reifies s W+       => BVar s Net+       -> BVar s (R 100)+       -> BVar s (R 5)+runNet net x = z+  where+    -- run first layer+    y = logistic $ (net ^^. nWeights1) #> x + (net ^^. nBias1)+    -- run second layer+    z = logistic $ (net ^^. nWeights2) #> y + (net ^^. nBias2)++logistic :: Floating a => a -> a+logistic x = 1 / (1 + exp (-x))+```++Note that we are using versions of `#>` lifted for `BVar`s, from the+*[hmatrix-backprop][]* library:++```haskell+(#>) :: BVar s (L m n) -> BVar s (R n) -> BVar s (R m)+```++[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop++#### Higher-Kinded Data Interface++Using the lens based interface, you can't directly pattern match and construct+fields.  To allow for directly pattern matching, there's another interface+option involving the "Higher-Kinded Data" techniques described in [this+article][hkd].++[hkd]: http://reasonablypolymorphic.com/blog/higher-kinded-data/++If we had a type-family (that can be re-used for all of your data types):++```haskell top+type family HKD f a where+    HKD Identity a = a+    HKD f        a = f a+```++We can define `Net` instead as:++```haskell top+data Met' f = M { _mWeights1 :: HKD f (L 20 100)+                , _mBias1    :: HKD f (R 20)+                , _mWeights2 :: HKD f (L  5  20)+                , _mBias2    :: HKD f (R  5)+                }+  deriving Generic+```++Then our *original* type is:++```haskell top+type Met = Met' Identity++deriving instance Show Met+instance Backprop Met+```++`Met` is the same as `Net` in every way -- it can be pattern matched on to get+the `L 20 100`, etc. (the `Identity` disappears):++```haskell top+getMetBias1 :: Met -> R 20+getMetBias1 (M _ b _ _) = b+```++The benefit of this is that we can now directly pattern match on a `BVar s Met`+to get the internal fields as `BVar`s using `splitBV` as a view pattern (or the+`BV` pattern synonym):++```haskell top+runMet :: Reifies s W+       => BVar s Met+       -> BVar s (R 100)+       -> BVar s (R 5)+runMet (splitBV -> M w1 b1 w2 b2) x = z+  where+    -- run first layer+    y = logistic $ w1 #> x + b1+    -- run second layer+    z = logistic $ w2 #> y + b2+```++Now, the `M w1 b1 w2 b2` pattern can be used to deconstruct *both* "normal"+`Met`s, as well as a `BVar s Met` (with `splitBV` or `BV`).++### Potential or Many Fields++Some values "may" or "may not" have values of a given field.  An example would+include the nth item in a list or vector, or the `Just` of a `Maybe`.++For these, the lens-based (prism-based/traversal-based) interface is the main way to access+partial fields.  You can use `(^^?)` or `previewVar` with any `Traversal`:++```haskell+(^?)  ::        a -> Traversal' a b -> Maybe         b+(^^?) :: BVar s a -> Traversal' a b -> Maybe (BVar s b)+```++If the value in the `BVar` "has" that field, then you'll get a `Just` with the+`BVar` of that field's contents.  If it doesn't, you'll get a `Nothing`.++You can use this with any prism or traversal, like using `_head` to get the+first item in a list if it exists.++If you have a type that might contain *many* values of a field (like a tree or+list), you can use `(^^..)` or `toListOfVar`, which works on any `Traversal`:++```haskell+(^..)  ::        a -> Traversal' a b -> [       b]+(^^..) :: BVar s a -> Traversal' a b -> [BVar s b]+```++This can be used to implement `sequenceVar`, actually:++```haskell+sequenceVar :: BVar s [a] -> [BVar s a]+sequenceVar xs = xs ^^.. traverse+```++### Tuples++The `T2` pattern synonym is provided, which allow you to pattern match on a+`BVar s (a, b)` to get a `BVar s a` and `BVar s b`.  The `T3` pattern is also+provided, which does the same thing for three-tuples.++Note that `T2` and `T2` are bidirectional pattern synonyms, and can be used to+construct as well as deconstruct.++Combining BVars+---------------++The following techniques can be used to "combine" `BVar`s:++### Foldable Containers++The "opposite" of `sequenceVar` is `collectVar`, which takes a foldable+container of `BVar`s and returns a `BVar` containing that foldable container of+contents:++```haskell+collectVar :: (Backprop a, Foldable t, Functor t, Reifies s W)+           => t (BVar s a)+           -> BVar s (t a)+```++### Constructors++Sometimes you would like to combine a bunch of `BVar`s into a `BVar` of+specific container or data type.++#### isoVar++The simplest way to do this is using the `isoVar`, `isoVar2`, etc. family of+functions:++```haskell+isoVar2+    :: (Backprop a, Backprop b, Backprop c, Reifies s W)+    => (a -> b -> c)+    -> (c -> (a, b))+    -> BVar s a+    -> BVar s b+    -> BVar s c+```++So if we had a type like:++```haskell top+data DoubleInt = DI Double Int+```++We can combine a `Double` and `Int` into a `DoubleInt` using `isoVar2`:++```haskell+isoVar2 DI (\(DI x y) -> (x,y))+    :: Reifies s W+    => BVar s Double+    -> BVar s Int+    -> BVar s DoubleInt+```++#### Higher-Kinded Data Interface++You can also use the ["Higher Kinded Data"][hkd] interface, as well.  For our+`Met` type above, you can use `joinBV`, or the `BV` pattern synonym:++```haskell top+makeMet :: Reifies s W+        => BVar s (L 20 100)+        -> BVar s (R 20)+        -> BVar s (L  5  20)+        -> BVar s (R  5)+        -> BVar s Met+makeMet w1 b1 w2 b2 = joinBV (M w1 b1 w2 b2)+```++### Modifying fields++If you just want to "set" a specific field, you can use the lens-based+interface with `(.~~)` or `setVar`.  For example, if we wanted to set the+`_nWeights2` field of a `Net` to a new matrix, we can do:++```haskell+myNet & nWeights2 .~~ newMatrix+```++or++```haskell+setVar nWeights2+    :: Reifies s W+    => BVar s (L 20 5)+    -> BVar s Net+    -> BVar s Net+```++Prelude Modules+---------------++Finally, the *Prelude.Backprop* module has a lot of your normal Prelude+functions "lifted" to work on `BVar`s of values.  For many situations, these+aren't necessary, and normal Prelude functions will work just fine on `BVar`s+of values (like `(.)`).  However, it does have some convenient functions, like+`minimum`, `foldl'`, `fmap`, `toList`, `fromIntegral`, `realToFrac`, etc.+lifted to work on `BVar`s.  This module is meant to be imported qualified.++Moving On+=========++Now that you know all about `BVar`s, you really can just **[jump into the+haddocks][haddock]** and start writing programs.  The next section of this+documentation is more details about **[the `Backprop` typeclass][class]**.++[class]: https://backprop.jle.im/04-the-backprop-typeclass.html+[haddock]: https://hackage.haskell.org/package/backprop

@@ -0,0 +1,132 @@+---+title: The Backprop Typeclass+---++The Backprop Typeclass+======================++```haskell top hide+{-# LANGUAGE FlexibleContexts #-}+{-# LANGUAGE DataKinds        #-}+{-# LANGUAGE DeriveGeneric    #-}++import           GHC.Generics (Generic)+import           GHC.TypeNats+import           Numeric.LinearAlgebra.Static (L, R)+import           Numeric.Backprop+import           Numeric.Backprop.Class+import qualified Data.Vector as V+```++Most of the functions in this module require a `Backprop` constraint on values+you wish to backpropagate.  Even if you manage to get around it for the most+part, `gradBP` (the actual function to compute gradients) requires it on both+the inputs and outputs.  Let's dig deeper into what it is, and how to define+instances.++The Class+---------++The typeclass contains three methods: `zero`, `add`, and `one`:++```haskell+class Backprop a where+    zero :: a -> a+    add  :: a -> a -> a+    one  :: a -> a+```++`zero` is "zero" in the verb sense -- it takes a value and "zeroes out" all+components.  For a vector, this means returning a zero vector of the same+shape.  For a list, this means replacing all of the items with zero and+returning a list of the same length.  `one` does the same thing but with one.+`add` is used to add together contributions in gradients, and is usually a+component-wise addition.++Instances are provided for most common data types where it makes sense.++Custom Instances+----------------++### Generics++When defining your own custom types, if your custom type is has *a single+constructor* where all fields are instances of `Backprop`,  then *GHC.Generics*+can be used to write your instances automatically:++```haskell top hide+instance Backprop (R n) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++instance (KnownNat n, KnownNat m) => Backprop (L n m) where+    zero = zeroNum+    add  = addNum+    one  = oneNum+```++```haskell top+data MyType = MkMyType Double [Float] (R 10) (L 20 10) (V.Vector Double)+  deriving Generic+```++Nice type.  Since it has a single constructor and all of its fields are already+`Backprop` instances, we can just write:++```haskell top+instance Backprop MyType+```++and now your type can be backpropagated!++### Common Patterns++For writing "primitive" `Backprop` instances (types that aren't product types),+you can use the provided "helpers" from the *Numeric.Backprop.Class* module.++If your type is a `Num` instance, you can use `zeroNum`, `addNum`, and+`oneNum`:++```haskell+instance Backprop Double where+    zero = zeroNum+    add  = addNum+    one  = oneNum+```++If your type is made using a `Functor` instance, you can use `zeroFunctor` and+`oneFunctor`:++```haskell+instance Backprop a => Backprop (V.Vector a) where+    zero = zeroFunctor+    add  = undefined        -- ??+    one  = oneFunctor+```++And if your type has an `IsList` instance, you can use `addIsList`:++```haskell+instance Backprop a => Backprop (V.Vector a) where+    zero = zeroFunctor+    add  = addIsList+    one  = oneFunctor+```++### Completely Custom++Completely custom instances are also possible; you just need to implement+`zero`, `add`, and `one` as they make sense for your type.  Just make sure that+you obey [the laws][laws] for sane behavior!++[laws]: http://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Class.html++Moving On+=========++At this point, feel free to **[jump into the haddocks][haddock]**, or read on+further for **[a list of applications and resources][applications]**.++[haddock]: https://hackage.haskell.org/package/backprop+[applications]: https://backprop.jle.im/05-applications.html

@@ -0,0 +1,48 @@+---+title: Applications and Resources+---++Applications and Resources+==========================++Congratulations!  You are now a *backprop* master.  Maybe you've even looked at+the [haddocks][haddock], which has the technical run-down of all of the+functions and types in this library.  Now what?++*   Check out my [Introducing the backprop library][intro] blog post where I+    announced the library to the world.  In it, I introduce the library by+    building and training a full artificial neural network with it, and use it+    to classify the famous MNIST handwritten digit data set.++*   If you want an even more high-level perspective and inspiration, check out+    my [A Purely Functional Typed Approach to Trainable Models][models] blog+    series, where I talk about how looking at modeling through the lens of+    differentiable programming with purely functional typed code can provide+    new insights and help you develop and train effective models.++*   While they are mostly re-phrasings of the two things above, I also have+    some [example projects as literate haskell files][lhs] on the github+    repository for the library.  These are also [rendered as pdfs][renders] for+    easier reading.++*   If you're doing anything with linear algebra, why not check out the+    *[hmatrix-backprop][]* library, which provides the "backprop-lifted"+    operations that all of the above examples rely on for linear algebra+    operations?++[haddock]: https://hackage.haskell.org/package/backprop+[intro]: https://blog.jle.im/entry/introducing-the-backprop-library.html+[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html+[lhs]: https://github.com/mstksg/backprop/blob/master/samples+[renders]: https://github.com/mstksg/backprop/tree/master/renders+[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop++This is the end of the "end-user" documentation for *backprop*!  The rest of+all you need to know to use the library is in the **[haddocks on+hackage][haddock]**.++However, if you are a library writer who wants to offer your users the ability+to backpropagate your library functions, let's move on to the **[library+maintainer's guide to equipping your library with backprop][equipping]**!++[equipping]: https://backprop.jle.im/06-equipping-your-library.html

@@ -0,0 +1,776 @@+---+title: Equipping your Library+---++Equipping your Library for Backprop+===================================++```haskell top hide+{-# LANGUAGE DataKinds          #-}+{-# LANGUAGE DeriveGeneric      #-}+{-# LANGUAGE FlexibleContexts   #-}+{-# LANGUAGE FlexibleInstances  #-}+{-# LANGUAGE StandaloneDeriving #-}+{-# LANGUAGE TemplateHaskell    #-}+{-# LANGUAGE TypeFamilies       #-}+{-# LANGUAGE ViewPatterns       #-}+++import           Data.Functor.Identity+import qualified Data.List+import           GHC.Generics (Generic)+import           GHC.TypeNats+import           Inliterate.Import+import           Lens.Micro+import           Lens.Micro.TH+import           Numeric.Backprop+import           Numeric.Backprop.Class+import           Numeric.LinearAlgebra.Static (L, R, konst)+import           System.Random+import qualified Data.Vector                  as V+import qualified Numeric.LinearAlgebra.Static as H+import qualified Numeric.LinearAlgebra        as HU+```++So you want your users to be able to use your numerical library with+*backprop*, huh?++This page is specifically for library authors who want to allow their users to+use their library operations and API with *backprop*.  End-users of the+*backprop* library should not have to worry about the contents of this page.++Equipping your library with backprop involves providing "backprop-aware"+versions of your library functions.  *In fact*, it is possible to make a+library fully by providing *only* backprop versions of your functions, since+you can use a backprop-aware function as a normal function with `evalBP`.+Alternatively, you can re-export all of your functions in a separate module with+"backprop-aware" versions.++Know Thy Types+--------------++The most significant effort will be in lifting your library's functions.  If+you have a function:++```haskell+myFunc :: a -> b+```++Then its lifted version would have type:++```haskell+myFunc :: Reifies s W => BVar s a -> BVar s b+```++That is, instead of a function directly taking an `a` and returning a `b`, it's+a function taking a `BVar` containing an `a`, and returning a `BVar` containing+a `b`.++Functions taking multiple arguments can be translated pretty straightforwardly:++```haskell+func1   ::                       a ->        b ->        c+func1BP :: Reifies s W => BVar s a -> BVar s b -> BVar s c+```++And also functions returning multiple arguments:++```haskell+func2   ::                       a -> (       b,        c)+func2BP :: Reifies s W => BVar s a -> (BVar s b, BVar s c)+```++It is recommended (for ease of use with `-XTypeApplications`) that `Reifies s+W` be the *final* constraint in all code you write.++Note that almost all operations involving `BVar`'d items require that the+contents have a `Backprop` instance.  Alternative API's to backprop that+require `Num` instances instead (or explicitly specified addition functions)+are available in *Numeric.Backprop.Num* and *Numeric.Backprop.Explicit*.++The Easy Way+------------++`BVar` based functions are just normal functions, so they can be applied+normally and passed as first-class values.  If possible, if you can *utilize*+functions that are already `BVar`'d/lifted, then you can just define your API+in terms of those lifted functions.  This is also how *users* are expected to+be able to use your library: just use the lifted functions you provide, in+order to make their own lifted functions using normal function application and+composition.++However, if no lifted primitive functions are available, then you do have to do+some legwork to provide information on gradient computation for your types.+Ideally, you would only need to do this for some minimal set of your+operations, and then define the rest of them in terms of the functions you have+already lifted.++Lifting operations manually+---------------------------++A `BVar s a -> BVar s b` really encodes two things:++1.  A `a -> b` (the actual function)+2.  A `a -> b -> a` (the "scaled gradient" function)++The documentation for [Numeric.Backprop.Op][op] gives detail about what these+entail, with rendered math and examples.++[op]: http://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Op.html++The second function requires some elaboration.  Let's say you are writing a+lifted version of your function \\(y = f(x)\\) (whose derivative is+\\(\frac{dy}{dx}\\)), and that your *final result* at the end of your computation+is \\(z = g(f(x))\\) (whose derivative is \\(\frac{dz}{dx}\\)).  In that case, because of the+chain rule, \\(\frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx}\\).++The scaled gradient is the function which, *given* \\(\frac{dy}{dz}\\), *returns*+\\(\frac{dz}{dx}\\). (that is, returns \\(\frac{dz}{dy} \frac{dy}{dx}\\)).++For example, for the mathematical operation \\(y = f(x) = x^2\\), then, considering+\\(z = g(f(x))\\), \\(\frac{dz}{dx} = \frac{dz}{dy} 2x\\).+In fact, for all functions taking and returning scalars (just normal single+numbers), \\(\frac{dz}{dx} = \frac{dz}{dy} f'(x)\\).++With that in mind, let's write our "squared" op:++```haskell top+square+    :: (Num a, Backprop a, Reifies s W)+    => BVar s a+    -> BVar s a+square = liftOp1 . op1 $ \x ->+    ( x^2              , \dzdy -> dzdy * 2 * x)+--    ^- actual result   ^- scaled gradient function+```++Keeping along the same pattern, for \\(y = f(x) = \sin(x)\\), then, considering \\(z+= g(f(x))\\), \\(\frac{dz}{dx} = \frac{dz}{dy} \cos(x)\\).  So, we have:++```haskell top+liftedSin+    :: (Floating a, Backprop a, Reifies s W)+    => BVar s a+    -> BVar s a+liftedSin = liftOp1 . op1 $ \x ->+    ( sin x, \dzdy -> dzdy * cos x )+```++In general, for functions that take and return scalars:++```haskell+liftedF+    :: (Reifies s W, Backprop a, Num a)+    => BVar s a+    -> BVar s a+liftedF = liftOp1 . op1 $ \x ->+    ( f x, \dzdy -> dzdy * dfdx x )+```++For an example of every single numeric function in base Haskell, see [the+source of Op.hs][opsource] for the `Op` definitions for every method in `Num`,+`Fractional`, and `Floating`.++[opsource]: https://github.com/mstksg/backprop/blob/a7651b4549048a3aca73c79c6fbe07c3e8ee500e/src/Numeric/Backprop/Op.hs#L646-L787++### Non-trivial example++A simple non-trivial example is `sumElements`, which we can define to take the+*hmatrix* library's `R n` type (an n-vector of `Double`).  In this case, we+have to think about \\(g(\mathrm{sum}(\mathbf{x}))\\).  In this case, the types+guide our thinking:++```haskell+sumElements           :: R n -> Double+sumElementsScaledGrad :: R n -> Double -> R n+```++The simplest way for me to do this personally is to just take it element by+element.++1.  *Write out the functions in question, in a simple example*++    In our case:++    *   \\(y = f(\langle a, b, c \rangle) = a + b + c\\)+    *   \\(z = g(y) = g(a + b + c)\\)++2.  *Identify the components in your gradient*++    In our case, we have to return a gradient \\(\langle \frac{\partial z}{\partial a},+    \frac{\partial z}{\partial b}, \frac{\partial z}{\partial c} \rangle\\).++3.  *Work out each component of the gradient until you start to notice a+    pattern*++    Let's start with \\(\frac{\partial z}{\partial a}\\).  We need to find+    \\(\frac{\partial z}{\partial a}\\) in terms of \\(\frac{dz}{dy}\\):++    *   Through the chain rule, \\(\frac{\partial z}{\partial a} =+        \frac{dz}{dy} \frac{\partial y}{\partial a}\\).+    *   Because \\(y = a + b + c\\), we know that \\(\frac{\partial y}{\partial+        a} = 1\\).+    *   Because \\(\frac{\partial y}{\partial a} = 1\\), we know that+        \\(\frac{\partial z}{\partial a} = \frac{dz}{dy} \times 1 =+        \frac{dz}{dy}\\).++    So, our expression of \\(\frac{\partial z}{\partial a}\\) in terms of+    \\(\frac{dz}{dy}\\) is simple -- it's simply \\(\frac{\partial z}{\partial+    a} = \frac{dz}{dy}\\).++    Now, let's look at \\(\frac{\partial z}{\partial b}\\).  We need to find+    \\(\frac{\partial z}{\partial b}\\) in terms of \\(\frac{dz}{dy}\\).++    *   Through the chain rule, \\(\frac{\partial z}{\partial b} =+        \frac{dz}{dy} \frac{\partial y}{\partial b}\\).+    *   Because \\(y = a + b + c\\), we know that \\(\frac{\partial y}{\partial+        b} = 1\\).+    *   Because \\(\frac{\partial y}{\partial b} = 1\\), we know that+        \\(\frac{\partial z}{\partial b} = \frac{dz}{dy} \times 1 =+        \frac{dz}{dy}\\).++    It looks like \\(\frac{\partial z}{\partial b} = \frac{\partial z}{\partial+    y}\\), as well.++    At this point, we start to notice a pattern.  We can apply the same logic+    to see that \\(\frac{\partial z}{\partial c} = \frac{dz}{dy}\\).++4.  *Write out the pattern*++    Extrapolating the pattern, \\(\frac{\partial z}{\partial q}\\), where+    \\(q\\) is *any* component, is always going to be a constant --+    \\(\frac{dz}{dy}\\).++So in the end:++```haskell top hide+instance Backprop (R n) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++instance (KnownNat n, KnownNat m) => Backprop (L n m) where+    zero = zeroNum+    add  = addNum+    one  = oneNum++sumElements :: KnownNat n => R n -> Double+sumElements = HU.sumElements . H.extract+```++```haskell top+liftedSumElements+    :: (KnownNat n, Reifies s W)+    => BVar s (R n)+    -> BVar s Double+liftedSumElements = liftOp1 . op1 $ \xs ->+    ( sumElements xs, \dzdy -> konst dzdy )  -- a constant vector+```++### Multiple-argument functions++Lifting multiple-argument functions is the same thing, except using `liftOp2`+and `op2`, or `liftOpN` and `opN`.++A `BVar s a -> BVar s b -> BVar s c` encodes two things:++1.  The actual `a -> b -> c`+2.  The scaled gradient, `a -> b -> c -> (a, b)`.++The `c` parameter of the scaled gradient is again \\(\frac{dz}{dy}\\), and the+final `(a,b)` is a tuple of \\(\frac{\partial z}{\partial x_1}\\) and+\\(\frac{\partial z}{\partial x_2}\\): how \\(\frac{dz}{dy}\\) affects both of+the inputs.++For a simple example, let's look at \\(x + y\\).  Working it out:++*   \\(y = f(x_1, x_2) = x_1 + x_2\\)+*   \\(z = g(f(x_1, x_2)) = g(x_1 + x_2)\\)+*   Looking first for \\(\frac{\partial z}{\partial x_1}\\) in terms of+    \\(\frac{dz}{dy}\\):+    *   \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial+        y}{\partial x_1}\\) (chain rule)+    *   From \\(y = x_1 + x_2\\), we see that \\(\frac{\partial y}{\partial+        x_1} = 1\\)+    *   Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \times 1+        = \frac{dz}{dy}\\).+*   Looking second for \\(\frac{\partial z}{\partial x_2}\\) in terms of+    \\(\frac{dz}{dy}\\):+    *   \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} \frac{\partial+        y}{\partial x_2}\\) (chain rule)+    *   From \\(y = x_1 + x_2\\), we see that \\(\frac{\partial y}{\partial+        x_2} = 1\\)+    *   Therefore, \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} \times 1+        = \frac{dz}{dy}\\).+*   Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy}\\), and also+    \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy}\\).++Putting it into code:++```haskell top+add :: (Num a, Backprop a, Reifies s W)+    => BVar s a+    -> BVar s a+    -> BVar s a+add = liftOp2 . op2 $ \x1 x2 ->+    ( x1 + x2, \dzdy -> (dzdy, dzdy) )+```++Let's try our hand at multiplication, or \\(x * y\\):++*   \\(y = f(x_1, x_2) = x_1 x_2\\)+*   \\(z = g(f(x_1, x_2)) = g(x_1 x_2)\\)+*   Looking first for \\(\frac{d\partial }{d\partial _1}\\) in terms of+    \\(\frac{dz}{dy}\\):+    *   \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial+        y}{\partial x_1}\\) (chain rule)+    *   From \\(y = x_1 x_2\\), we see that \\(\frac{\partial y}{\partial x_1}+        = x_2\\)+    *   Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} x_2\\).+*   Looking second for \\(\frac{\partial z}{\partial x_2}\\) in terms of+    \\(\frac{dz}{dy}\\):+    *   \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} \frac{\partial+        y}{\partial x_1}\\) (chain rule)+    *   From \\(y = x_1 x_2\\), we see that \\(\frac{\partial y}{\partial x_2}+        = x_1\\)+    *   Therefore, \\(\frac{\partial z}{\partial x_2} = \frac{dz}{dy} x_1\\).+*   Therefore, \\(\frac{\partial z}{\partial x_1} = \frac{dz}{dy} x_2\\), and+    \\(\frac{\partial z}{\partial x_2} = x_1 \frac{dz}{dy}\\).++In code:++```haskell top+mul :: (Num a, Backprop a, Reifies s W)+    => BVar s a+    -> BVar s a+    -> BVar s a+mul = liftOp2 . op2 $ \x1 x2 ->+    ( x1 * x2, \dzdy -> (dzdy * x2, x1 * dzdy) )+```++For non-trivial examples involving linear algebra, see the source for the *[hmatrix-backprop][]* library.++[hmatrix-backprop]: http://hackage.haskell.org/package/hmatrix-backprop++Some examples, for the dot product between two vectors and for matrix-vector+multiplication:++```haskell top+-- import qualified Numeric.LinearAlgebra.Static as H++-- | dot product between two vectors+dot+    :: (KnownNat n, Reifies s W)+    => BVar s (R n)+    -> BVar s (R n)+    -> BVar s Double+dot = liftOp2 . op2 $ \u v ->+    ( u `H.dot` v+    , \dzdy -> (H.konst dzdy * v, u * H.konst dzdy)+    )+++-- | matrix-vector multiplication+(#>)+    :: (KnownNat m, KnownNat n, Reifies s W)+    => BVar s (L m n)+    -> BVar s (R n)+    -> BVar s (R m)+(#>) = liftOp2 . op2 $ \mat vec ->+    ( mat H.#> vec+    , \dzdy -> (dzdy `H.outer` vec, H.tr mat H.#> dzdy)+    )+```++### Returning multiple items++You can return tuples inside `BVar`s:++```haskell top+splitAt+    :: (Backprop a, Reifies s W)+    => Int+    -> BVar s [a]+    -> BVar s ([a], [a])+splitAt n = liftOp1 . op1 $ \xs ->+    let (ys, zs) = Data.List.splitAt n xs+    in  ((ys, zs), \(dys,dzs) -> dys ++ dzs)+                      -- assumes dys and dzs have the same lengths as ys and zs+```++This works as expected.  However, it is recommended, for the benefit of your+users, that you return a tuple of `BVar`s instead of a `BVar` of tuples:++```haskell top+splitAt'+    :: (Backprop a, Reifies s W)+    => Int+    -> BVar s [a]+    -> (BVar s [a], BVar s [a])+splitAt' n xs = (yszs ^^. _1, yszs ^^. _2)+  where+    yszs = liftOp1 (op1 $ \xs' ->+        let (ys, zs) = Data.List.splitAt n xs'+        in  ((ys, zs), \(dys,dzs) -> dys ++ dzs)+      ) xs+```++using `_1` and `_2` from the *[microlens][]* or *[lens][]* packages.  This+might also be cleaner if you take advantage of the `T2` or `T3` pattern+synonyms:++[microlens]: http://hackage.haskell.org/package/microlens+[lens]: http://hackage.haskell.org/package/lens++```haskell top+splitAt''+    :: (Backprop a, Reifies s W)+    => Int+    -> BVar s [a]+    -> (BVar s [a], BVar s [a])+splitAt'' n xs = (ys, zs)+  where+    T2 ys zs = liftOp1 (op1 $ \xs' ->+        let (ys, zs) = Data.List.splitAt n xs'+        in  ((ys, zs), \(dys,dzs) -> dys ++ dzs)+      ) xs+```++### Isomorphisms++If your function witnesses an isomorphism, there are handy combinators for+making this easy to write.  This is especially useful in the case of data+constructors:++```haskell top+newtype Foo = MkFoo { getFoo :: Double }+  deriving Generic++instance Backprop Foo++mkFoo+    :: Reifies s W+    => BVar s Double+    -> BVar s Foo+mkFoo = isoVar MkFoo getFoo++data Bar = MkBar { bar1 :: Double, bar2 :: Float }+  deriving Generic++instance Backprop Bar++mkBar+    :: Reifies s W+    => BVar s Double+    -> BVar s Float+    -> BVar s Bar+mkBar = isoVar2 MkBar (\b -> (bar1 b, bar2 b))+```++Note also that if you have a newtype with one constructor (or any other two+`Coercible` types), you can simply use `coerceVar`:++```haskell top+mkFoo'+    :: BVar s Double+    -> BVar s Foo+mkFoo' = coerceVar          -- requires no `Reifies s W` constraint+```++### NoGrad++If you do decide to go to the extreme, and provide *only* a BVar-based+interface to your library (and no non-BVar based one), then you might have a+situation where you have a function where you cannot define the gradient --+maybe no gradient exists, or you haven't put in the time to write one.  In this+case, you can use `noGrad` and `noGrad1`:++```haskell top+negateNoGrad+    :: (Num a, Backprop a, Reifies s W)+    => BVar s a+    -> BVar s a+negateNoGrad = liftOp1 (noGrad1 negate)+```++This function can still be used with `evalBP` to get the correct answer.  It+can even be used with `gradBP` if the result is never used in the final answer.++However, if it *is* used in the final answer, then computing the gradient will+throw a runtime exception.++Be sure to warn your users!  Like any partial function, this is not recommended+unless in extreme circumstances.++Monadic Operations+------------------++This should all work if your operations are all "pure".  However, what about+the cases where your operations have to be performed in some Applicative or+Monadic context?++For example, what if `add :: X -> X -> IO X` ?++One option you can do is to newtype-wrap your operations, and then give those a+backprop instance:++```haskell top hide+data X++zeroForX :: X -> X+zeroForX = undefined+addForX  :: X -> X -> IO X+addForX = undefined+oneForX :: X -> X+oneForX = undefined+```++```haskell top+newtype IOX = IOX (IO X)++instance Backprop IOX where+    zero (IOX x) = IOX (fmap zeroForX x)+    -- or, depending on the type of `zeroForX`:+    -- zero (IOX x) = IOX (zeroForX =<< x)++    add (IOX x) (IOX y) = IOX $ do+      x' <- x+      y' <- y+      addForX x' y'++    one (IOX x) = IOX (fmap oneForX x)+```++And you can define your functions in terms of this:++```haskell top+addX+    :: Reifies s W+    => BVar s IOX+    -> BVar s IOX+    -> BVar s IOX+addX = liftOp2 . op2 $ \(IOX x) (IOX y) ->+    ( IOX (do x' <- x; y' <- y; addForX x' y')+    , \dzdy -> (dzdy, dzdy)+    )+```++This should work fine as long as you never "branch" on any *results* of your+actions.  You must not ever need to peek inside the *results* of the action in+order to decide *what* operations to do next.  In other words, this works if+the operations you need to perform are all known and fixed before-hand, before+any actions are performed.  So, this means no access to the `Eq` or `Ord`+instances of BVars (unless your monad has `Eq` or `Ord` instances defined).++A newtype wrapper is provided to give you this behavior automatically -- it's+`ABP`, from *Numeric.Backprop* and *Numeric.Backprop.Class*.++```haskell+type IOX = ABP IO X+```++However, this will not work if you need to do things like compare contents,+etc. to decide what operations to use.++At the moment, this is not supported.  Please open an issue if this becomes an+issue!++Supporting Data Types+---------------------++Your library will probably have data types that you expect your users to use.+To equip your data types for backpropagation, you can take a few steps.++### Backprop Class++First of all, all of your library's types should have instances of the+[`Backprop` typeclass][class].  This allows values of your type to be used in+backpropagatable functions.  See the [Backprop typeclass section][tcdocs] of+this documentation for more information on writing a `Backprop` instance for+your types.++[class]: https://hackage.haskell.org/package/backprop/docs/Numeric-Backprop-Class.html+[tcdocs]: https://backprop.jle.im/04-the-backprop-typeclass.html++In short:++1.  If your type is a type with a single constructor whose fields are all+    instances of `Backprop`, you can just write `instance Backprop MyType`, and+    the instance is generated automatically (as long as your type has a+    `Generic` instance)++    ```haskell top+    data MyType = MkMyType Double [Float] (R 10) (L 20 10) (V.Vector Double)+      deriving Generic++    instance Backprop MyType+    ```++2.  If your type is an instance of `Num`, you can use `zeroNum`, `addNum`, and+    `oneNum` to get free definitions of the typeclass methods.++    ```haskell+    instance Backprop Double where+        zero = zeroNum+        add  = addNum+        one  = oneNum+    ```++3.  If your type is made using a `Functor` instance, you can use `zeroFunctor`+    and `oneFunctor`:++    ```haskell+    instance Backprop a => Backprop (V.Vector a) where+        zero = zeroFunctor+        add  = undefined        -- ??+        one  = oneFunctor+    ```++4.  If your type has an `IsList` instance, you can use `addIsList`:++    ```haskell+    instance Backprop a => Backprop (V.Vector a) where+        zero = zeroFunctor+        add  = addIsList+        one  = oneFunctor+    ```++For more details, see the [aforementioned documentation][tcdocs] or the [actual+typeclass haddock documentation][class].++### Accessors++If you have product types, users should be able to access values inside `BVar`s+of your data type.  There are two main ways to provide access: the lens-based+interface and the higher-kinded-data-based interface.++The lens-based interface gives your users "getter" and "setter" functions for+fields, and the higher-kinded-data-based interface lets your users pattern+match on your data type's original constructor to get fields and construct+values.++#### Lens-Based Interface++If you are defining a product type, like++```haskell top+data MyType = MT { _mtDouble  :: Double+                 , _mtInt     :: Int+                 , _mtDoubles :: [Double]+                 }+```++Users who have a `BVar s MyType` can't normally access the fields inside,+because you can't directly pattern match normally, and the record accessors+are `MyType -> Int` (unlifted).  As a library maintainer, you can provide them+*lenses* to the fields, either generated automatically using the *[lens][]* or+*[microlens-th][]* packages:++[lens]: http://hackage.haskell.org/package/lens+[microlens-th]: http://hackage.haskell.org/package/microlens-th++```haskell top+makeLenses ''MyType+```++or manually by hand:++```haskell top+mtInt' :: Functor f => (Int -> f Int) -> MyType -> f MyType+mtInt' f mt = (\i -> mt { _mtInt = i }) <$> f (_mtInt mt)+```++Now, users can use `^.` or `view` from the *lens* or *[microlens][]* packages+to retrieve your fields:++[microlens]: http://hackage.haskell.org/package/microlens++```haskell+(^. mtDouble)  ::        MyType ->        Double+```++And `(^^.)` and `viewVar` from *backprop* to retrieve fields from a `BVar`:++```haskell+(^^. mtDouble) :: BVar s MyType -> BVar s Double+```++They can also use `set` or `.~` to modify fields, and `setVar` and `.~~` to+modify and "set" fields in a `BVar`:++```haskell+set    mtDouble ::        Double ->        MyType ->        MyType+setVar mtDouble :: BVar s Double -> BVar s MyType -> BVar s MyType+```++#### Higher-Kinded Data Interface++The alternative "Higher-Kinded Data" technique, inspired by [this+article][hkd], allows your users to directly pattern match on `BVar`s of your+types to get their contents.++[hkd]: http://reasonablypolymorphic.com/blog/higher-kinded-data/++Doing this requires modifying the definition of your data types slightly.+Instead of `MyType` above, we can make a type family that can be re-used for+all of your data types:++```haskell top+type family HKD f a where+    HKD Identity a = a+    HKD f        a = f a+```++and define your data types in terms of this type family (remembering to derive+`Generic`):++```haskell top+data MyType2' f = MT2 { mt2Double  :: HKD f Double+                      , mt2Int     :: HKD f Int+                      , mt2Doubles :: HKD f [Double]+                      }+  deriving Generic+```++Now your original data type can be recovered with `MyType2' Identity`, and can+be pattern matched directly in the same way as the original type (the+`Identity` disappears):++```haskell top+type MyType2 = MyType2' Identity++deriving instance Show MyType2+instance Backprop MyType2++getMT2Double :: MyType2 -> Double+getMT2Double (MT2 d _ _) = d+```++But now, users can *pattern match* on a `BVar s MyType2` to get `BVar`s of the+contents, with `splitBV` or the `BV` pattern synonym:++```haskell top+getMT2DoubleBVar+    :: Reifies s W+    => BVar s MyType2+    -> BVar s Double+getMT2DoubleBVar (splitBV -> MT2 d _ _) = d+```++Under `splitBV`, your users can pattern match on the `MT2` constructor and get+the contents as `BVar`s.++Users can also use `joinBV` (or the `BV` pattern synonym in constructor mode)+to re-construct a `BVar` of `MyType2` in terms of `BVar`s of its contents using+the `MT2` constructor:++```haskell top+makeMyType2+    :: Reifies s W+    => BVar s Double+    -> BVar s Int+    -> BVar s [Double]+    -> BVar s MyType2+makeMyType2 d i ds = joinBV $ MT2 d i ds+```+

@@ -0,0 +1,61 @@+---+title: Home+---++Welcome to Backprop+===================++Automatic *heterogeneous* back-propagation.++*Write your functions normally* to compute your result, and the library will+*automatically compute your gradient*!++```haskell top hide+import           Numeric.Backprop+```++```haskell eval+gradBP (\x -> x^2 + 3) (9 :: Double)+```++Differs from [ad][] by offering full heterogeneity -- each intermediate step+and the resulting value can have different types (matrices, vectors, scalars,+lists, etc.)++[ad]: http://hackage.haskell.org/package/ad++```haskell eval+gradBP2 (\x xs -> sum (map (**2) (sequenceVar xs)) / x)+        (9       :: Double  )+        ([1,6,2] :: [Double])+```++Useful for applications in [differential programming][dp] and deep learning for+creating and training numerical models, especially as described in this blog+post on [a purely functional typed approach to trainable models][models].+Overall, intended for the implementation of gradient descent and other numeric+optimization techniques.  Comparable to the python library [autograd][].++[dp]: https://www.facebook.com/yann.lecun/posts/10155003011462143+[models]: https://blog.jle.im/entry/purely-functional-typed-models-1.html+[autograd]: https://github.com/HIPS/autograd++**[Get started][getting started]** with the introduction and walkthrough!  Full+technical documentation is also **[available on hackage][hackage]** if you want+to skip the introduction and get right into using the library.  Support is+available on the **[gitter channel][gitter]**!++[getting started]: https://backprop.jle.im/01-getting-started.html++[hackage]: http://hackage.haskell.org/package/backprop+[gitter]: https://gitter.im/haskell-backprop/Lobby++[![Join the chat at https://gitter.im/haskell-backprop/Lobby](https://badges.gitter.im/haskell-backprop/Lobby.svg)](https://gitter.im/haskell-backprop/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)++[![Beerpay](https://beerpay.io/mstksg/backprop/badge.svg?style=beer-square)](https://beerpay.io/mstksg/backprop)++[![backprop on Hackage](https://img.shields.io/hackage/v/backprop.svg?maxAge=86400)](https://hackage.haskell.org/package/backprop)+[![backprop on Stackage LTS 11](http://stackage.org/package/backprop/badge/lts-11)](http://stackage.org/lts-11/package/backprop)+[![backprop on Stackage Nightly](http://stackage.org/package/backprop/badge/nightly)](http://stackage.org/nightly/package/backprop)+[![Build Status](https://travis-ci.org/mstksg/backprop.svg?branch=master)](https://travis-ci.org/mstksg/backprop)+

@@ -1,3 +1,4 @@+{-# LANGUAGE CPP              #-} {-# LANGUAGE DataKinds        #-} {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs            #-}@@ -49,8 +50,8 @@ -- reading the docs for 'BVar'. -- -- If you are writing a library, see--- <https://github.com/mstksg/backprop/wiki/Equipping-your-Library-with-Backprop>--- for a guide for equipping your library with backpropatable operations.+-- <https://backprop.jle.im/06-equipping-your-library.html> for a guide for+-- equipping your library with backpropatable operations. -- -- In the original version 0.1, this module required 'Num' instances for -- methods instead of 'Backprop' instances.  This interface is still@@ -90,6 +91,7 @@     -- $hkd   , splitBV   , joinBV+  , pattern BV   , E.BVGroup     -- * 'Op'   , Op(..)@@ -111,6 +113,7 @@   , Reifies   ) where +import           Data.Bifunctor import           Data.Maybe import           Data.Reflection import           Data.Type.Index@@ -195,20 +198,21 @@     => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as     -> (b, Tuple as)-backpropN = E.backpropN E.zeroFuncs E.oneFunc+backpropN f = second ($ E.oneFunc) . E.backpropN E.zeroFuncs f {-# INLINE backpropN #-}  -- | 'backpropN', but allows you to provide the gradient of the "final -- result" with respect to the output of your function.  See 'backpropWith' -- for more details.+-- +-- Note that argument order changed in v0.2.3. -- -- @since 0.2.0.0 backpropWithN     :: (Every Backprop as, Known Length as)     => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, Tuple as)+    -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function backpropWithN = E.backpropWithN E.zeroFuncs {-# INLINE backpropWithN #-} @@ -224,7 +228,7 @@     => (forall s. Reifies s W => BVar s a -> BVar s b)     -> a     -> (b, a)-backprop = E.backprop E.zeroFunc E.oneFunc+backprop f = second ($ E.oneFunc) . E.backprop E.zeroFunc f {-# INLINE backprop #-}  -- | A version of 'backprop' that allows you to specify the gradent of your@@ -241,13 +245,14 @@ -- 'backprop' is essentially 'backpropWith' with @'const' 1@ for scalars -- and 'Num' instances. --+-- Note that argument order changed in v0.2.3+-- -- @since 0.2.0.0 backpropWith     :: Backprop a     => (forall s. Reifies s W => BVar s a -> BVar s b)     -> a-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, a)+    -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function backpropWith = E.backpropWith E.zeroFunc {-# INLINE backpropWith #-} @@ -296,21 +301,22 @@     -> a     -> b     -> (c, (a, b))-backprop2 = E.backprop2 E.zeroFunc E.zeroFunc E.oneFunc+backprop2 f x = second ($ E.oneFunc) . E.backprop2 E.zeroFunc E.zeroFunc f x {-# INLINE backprop2 #-}  -- | 'backprop2', but allows you to provide the gradient of the "final -- result" with respect to the output of your function.  See 'backpropWith' -- for more details. --+-- Note that argument order changed in v0.2.3+-- -- @since 0.2.0.0 backpropWith2     :: (Backprop a, Backprop b)     => (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c)     -> a     -> b-    -> (c -> c)                 -- ^ Gradient of final result with respect to output of function-    -> (c, (a, b))+    -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function backpropWith2 = E.backpropWith2 E.zeroFunc E.zeroFunc {-# INLINE backpropWith2 #-} @@ -559,11 +565,13 @@ -- gradient is assumed to correspond with the second item in the input, -- etc.; this can cause unexpected behavior in 'Foldable' instances that -- don't have a fixed number of items.+--+-- Prior to v0.2.3, required a 'Backprop' constraint on @t a@. collectVar-    :: (Foldable t, Functor t, Backprop a, Backprop (t a), Reifies s W)+    :: (Foldable t, Functor t, Backprop a, Reifies s W)     => t (BVar s a)     -> BVar s (t a)-collectVar = E.collectVar E.addFunc E.zeroFunc E.zeroFunc+collectVar = E.collectVar E.addFunc E.zeroFunc {-# INLINE collectVar #-}  -- | Lift an 'Op' with an arbitrary number of inputs to a function on the@@ -715,7 +723,9 @@ pattern T2 x y <- (\xy -> (xy ^^. _1, xy ^^. _2) -> (x, y))   where     T2 = isoVar2 (,) id-{-# COMPLETE T2 #-}+#if MIN_VERSION_base(4,10,0)+{-# COMPLETE BV #-}+#endif  -- | Useful pattern for constructing and deconstructing 'BVar's -- three-tuples.@@ -730,7 +740,9 @@ pattern T3 x y z <- (\xyz -> (xyz ^^. _1, xyz ^^. _2, xyz ^^. _3) -> (x, y, z))   where     T3 = isoVar3 (,,) id-{-# COMPLETE T3 #-}+#if MIN_VERSION_base(4,10,0)+{-# COMPLETE BV #-}+#endif  -- $hkd --@@ -786,6 +798,13 @@ -- myFunction ('splitBV' -> MT x y) =  x + 'Prelude.Backprop.sum' y -- @ --+-- Or also, using the 'BV' pattern synonym:+--+-- @+-- myFunction :: 'BVar' s MyType -> BVar s Double+-- myFunction ('BV' (MT x y)) =  x + 'Prelude.Backprop.sum' y+-- @+-- -- If you use 'splitBV', the contents will be a @BVar s Double@ and a @BVar -- s [Double]@.  It lets you "extract" the fields, because your 'MyType'' -- constructor now holds a @'BVar' s Double@ and a @BVar s [Double]@,@@ -805,6 +824,9 @@ -- myOtherFunction x y = 'joinBV' $ MT x y -- @ --+-- The 'BV' pattern synonym abstracts over manual application of 'splitBV'+-- and 'joinBV' as a pattern.+-- -- This will work with all data types made with a single constructor, whose -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'.@@ -821,6 +843,9 @@ -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'.  The type also must derive 'Generic'. --+-- Note that 'BV' is a pattern synonym version where the deconstructor is+-- exactly a view into 'splitBV'.+-- -- @since 0.2.2.0 splitBV     :: ( Generic (z f)@@ -836,7 +861,7 @@ splitBV = E.splitBV E.addFunc E.addFuncs E.zeroFunc E.zeroFuncs {-# INLINE splitBV #-} --- | Split out a 'BVar' of "higher-kinded data type", a la+-- | Assemble a 'BVar' of "higher-kinded data type", a la -- <http://reasonablypolymorphic.com/blog/higher-kinded-data/> -- -- It lets you take a 'BVar' of every field of a value, and join them into@@ -848,6 +873,9 @@ -- fields are all instances of 'Backprop', where the type itself has an -- instance of 'Backprop'. --+-- Note that 'BV' is a pattern synonym version where the constructor is+-- exactly 'joinBV'.+-- -- @since 0.2.2.0 joinBV     :: ( Generic (z f)@@ -862,3 +890,31 @@     -> BVar s (z f)         -- ^ 'BVar' of combined value joinBV = E.joinBV E.addFunc E.addFuncs E.zeroFunc E.zeroFuncs {-# INLINE joinBV #-}++-- | Pattern synonym wrapping manual usage of 'splitBV' and 'joinBV'.  It+-- is a pattern for a @'BVar' s (z f)@ containing a @z ('BVar' s)@+--+-- @since 0.2.3.0+pattern BV+    :: ( Generic (z f)+       , Generic (z (BVar s))+       , E.BVGroup s as (Rep (z f)) (Rep (z (BVar s)))+       , Backprop (Rep (z f) ())+       , Backprop (z f)+       , Every Backprop as+       , Known Length as+       , Reifies s W+       )+#if MIN_VERSION_base(4,10,0)+    => z (BVar s)           -- ^ 'BVar's of fields+    -> BVar s (z f)         -- ^ 'BVar' of combined value+#else+    => z (BVar s)+    -> BVar s (z f)+#endif+pattern BV v <- (splitBV->v)+  where+    BV = joinBV+#if MIN_VERSION_base(4,10,0)+{-# COMPLETE BV #-}+#endif

@@ -165,35 +165,40 @@ {-# INLINE auto #-}  -- | 'Numeric.Backprop.backpropWithN', but with explicit 'zero'.+--+-- Note that argument order changed in v0.2.3.+--+-- @since 0.2.0.0 backpropWithN     :: Prod ZeroFunc as     -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, Tuple as)-backpropWithN zfs f xs g = backpropN zfs (OF g) f xs+    -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function+backpropWithN zfs f = second (. OF) . backpropN zfs f {-# INLINE backpropWithN #-}  -- | 'Numeric.Backprop.backprop', but with explicit 'zero' and 'one'.+--+-- Note that argument order changed in v0.2.3. backprop     :: ZeroFunc a-    -> OneFunc b     -> (forall s. Reifies s W => BVar s a -> BVar s b)     -> a-    -> (b, a)-backprop zfa ofb f = second (getI . head')-                   . backpropN (zfa :< Ø) ofb (f . head')-                   . only_+    -> (b, OneFunc b -> a)+backprop zfa f = second ((getI . head') .)+               . backpropN (zfa :< Ø) (f . head')+               . only_ {-# INLINE backprop #-}  -- | 'Numeric.Backprop.backpropWith', but with explicit 'zero'.+--+-- Note that argument order changed in v0.2.3. backpropWith     :: ZeroFunc a     -> (forall s. Reifies s W => BVar s a -> BVar s b)     -> a-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, a)-backpropWith zfa f x g = backprop zfa (OF g) f x+    -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function+backpropWith zfa f = second (. OF) . backprop zfa f {-# INLINE backpropWith #-}  -- | 'evalBP' but with no arguments.  Useful when everything is just given@@ -221,7 +226,7 @@     -> (forall s. Reifies s W => BVar s a -> BVar s b)     -> a     -> a-gradBP zfa ofb f = snd . backprop zfa ofb f+gradBP zfa ofb f = ($ ofb) . snd . backprop zfa f {-# INLINE gradBP #-}  -- | 'Numeric.Backprop.gradBP', Nbut with explicit 'zero' and 'one'.@@ -231,34 +236,38 @@     -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as     -> Tuple as-gradBPN zfas ofb f = snd . backpropN zfas ofb f+gradBPN zfas ofb f = ($ ofb) . snd . backpropN zfas f {-# INLINE gradBPN #-}  -- | 'Numeric.Backprop.backprop2', but with explicit 'zero' and 'one'.+--+-- Note that argument order changed in v0.2.3. backprop2     :: ZeroFunc a     -> ZeroFunc b-    -> OneFunc c     -> (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c)     -> a     -> b-    -> (c, (a, b))-backprop2 zfa zfb ofc f x y = second (\(dx ::< dy ::< Ø) -> (dx, dy)) $-    backpropN (zfa :< zfb :< Ø) ofc+    -> (c, OneFunc c -> (a, b))+backprop2 zfa zfb f x y = second ((\(dx ::< dy ::< Ø) -> (dx, dy)) .) $+    backpropN (zfa :< zfb :< Ø)         (\(x' :< y' :< Ø) -> f x' y')         (x ::< y ::< Ø) {-# INLINE backprop2 #-}  -- | 'Numeric.Backprop.backpropWith2', but with explicit 'zero'.+--+-- Note that argument order changed in v0.2.3.+--+-- @since 0.2.0.0 backpropWith2     :: ZeroFunc a     -> ZeroFunc b     -> (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c)     -> a     -> b-    -> (c -> c)                 -- ^ Gradient of final result with respect to output of function-    -> (c, (a, b))-backpropWith2 zfa zfb f x y g = backprop2 zfa zfb (OF g) f x y+    -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function+backpropWith2 zfa zfb f x = second (. OF) . backprop2 zfa zfb f x {-# INLINE backpropWith2 #-}  -- | 'evalBP' for a two-argument function.  See@@ -281,7 +290,7 @@     -> a     -> b     -> (a, b)-gradBP2 zfa zfb ofc f x = snd . backprop2 zfa zfb ofc f x+gradBP2 zfa zfb ofc f x = ($ ofc) . snd . backprop2 zfa zfb f x {-# INLINE gradBP2 #-}  -- | 'Numeric.Backprop.isoVar' with explicit 'add' and 'zero'.

@@ -171,8 +171,8 @@ -- using 'Numeric.Backprop.liftOp' and related functions.  This is how you -- can create primitive functions that users can use to manipulate your -- library's values.  See--- <https://github.com/mstksg/backprop/wiki/Equipping-your-Library-with-Backprop>--- for a detailed guide.+-- <https://backprop.jle.im/06-equipping-your-library.html> for a detailed+-- guide. -- -- For example, the /hmatrix/ library has a matrix-vector multiplication -- function, @#> :: L m n -> R n -> L m@.@@ -411,7 +411,7 @@ liftOp3 afa afb afc z o !v !u !w = unsafePerformIO $ liftOp3_ afa afb afc z o v u w {-# INLINE liftOp3 #-} --- TODO: can we get the zero and scale func from the bvar?+-- TODO: can we get the zero and add func from the bvar? viewVar_     :: forall a b s. Reifies s W     => AddFunc a@@ -438,7 +438,7 @@ viewVar af z l !v = unsafePerformIO $ viewVar_ af z l v {-# INLINE viewVar #-} --- TODO: can zero and scale func be gotten from the input bvars?+-- TODO: can zero and add func be gotten from the input bvars? setVar_     :: forall a b s. Reifies s W     => AddFunc a@@ -484,15 +484,14 @@ sequenceVar af z !v = unsafePerformIO $ traverseVar' af z id traverse v {-# INLINE sequenceVar #-} --- TODO: can scale funcs and zeros be had from bvars and Functor instance?+-- TODO: can add funcs and zeros be had from bvars and Functor instance? collectVar_     :: forall t a s. (Reifies s W, Foldable t, Functor t)     => AddFunc a     -> ZeroFunc a-    -> ZeroFunc (t a)     -> t (BVar s a)     -> IO (BVar s (t a))-collectVar_ af z z' !vs = withV (toList vs) $ \(vVec :: Vec n (BVar s a)) -> do+collectVar_ af z !vs = withV (toList vs) $ \(vVec :: Vec n (BVar s a)) -> do     let tn :: TapeNode (t a)         tn = TN           { _tnInputs = vecToProd (vmap ((`IR` runAF af) . getI) vVec)@@ -501,18 +500,27 @@                       . toList           }     traverse_ (evaluate . forceBVar) vs-    insertNode tn (_bvVal <$> vs) z' (reflect (Proxy @s))+    insertNode tn (_bvVal <$> vs) (ZF $ fmap (runZF z)) (reflect (Proxy @s)) {-# INLINE collectVar_ #-}  -- | 'Numeric.Backprop.collectVar', but with explicit 'add' and 'zero'.+--+-- NOTE: Prior to v0.2.3, this required an extra @'ZeroFunc' (t a)@ input.+-- However, after v0.2.3, the 'ZeroFunc' is now derived from the 'Functor'+-- instance of @t@.  This makes the API a little more convenient, and it+-- enforces consistency with the @'ZeroFunc' a@, so people can't pass in+-- nonsense combinations.+--+-- Please submit an issue to the issue tracker if you find yourself in+-- a situation where you need the flexibility to provide a separte+-- @'ZeroFunc' a@ and @'ZeroFunc' (t a)@. collectVar     :: forall t a s. (Reifies s W, Foldable t, Functor t)     => AddFunc a     -> ZeroFunc a-    -> ZeroFunc (t a)     -> t (BVar s a)     -> BVar s (t a)-collectVar af z z' !vs = unsafePerformIO $ collectVar_ af z z' vs+collectVar af z !vs = unsafePerformIO $ collectVar_ af z vs {-# INLINE collectVar #-}  traverseVar'@@ -613,18 +621,19 @@ {-# INLINE gradRunner #-}  -- | 'Numeric.Backprop.backpropN', but with explicit 'zero' and 'one'.+--+-- Note that argument order changed in v0.2.3. backpropN     :: forall as b. ()     => Prod ZeroFunc as-    -> OneFunc b     -> (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as-    -> (b, Tuple as)-backpropN zfs ofb f !xs = (y, g)+    -> (b, OneFunc b -> Tuple as)+backpropN zfs f !xs = (y, g)   where     !(!tp@(!_,!_),!y) = unsafePerformIO $ fillWengert f xs-    g :: Tuple as-    g = runST $ do+    g :: OneFunc b -> Tuple as+    g ofb = runST $ do         r <- initRunner tp $ bimap getSum (`appEndo` [])                            . fst                            $ zipWithPM_ go zfs xs

@@ -88,6 +88,7 @@   , Reifies   ) where +import           Data.Bifunctor import           Data.Maybe import           Data.Reflection import           Data.Type.Index@@ -114,19 +115,22 @@     => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as     -> (b, Tuple as)-backpropN = E.backpropN E.zfNums E.ofNum+backpropN f = second ($ E.ofNum) . E.backpropN E.zfNums f {-# INLINE backpropN #-}  -- | 'Numeric.Backprop.backpropWithN', but with 'Num' constraints instead -- of 'Backprop' constraints. -- -- See 'backpropN' for information on the 'Every' constraint.+--+-- Note that argument order changed in v0.2.3.+--+-- @since 0.2.0.0 backpropWithN     :: (Every Num as, Known Length as)     => (forall s. Reifies s W => Prod (BVar s) as -> BVar s b)     -> Tuple as-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, Tuple as)+    -> (b, (b -> b) -> Tuple as) -- ^ Takes function giving gradient of final result given the output of function backpropWithN = E.backpropWithN E.zfNums {-# INLINE backpropWithN #-} @@ -140,7 +144,7 @@     => (forall s. Reifies s W => BVar s a -> BVar s b)     -> a     -> (b, a)-backprop = E.backprop E.zfNum E.ofNum+backprop f = second ($ E.ofNum) . E.backprop E.zfNum f {-# INLINE backprop #-}  -- | 'Numeric.Backprop.backpropWith', but with 'Num' constraints instead of@@ -148,12 +152,15 @@ -- -- See module documentation for "Numeric.Backprop.Num" for information on -- using this with tuples.+--+-- Note that argument order changed in v0.2.3.+--+-- @since 0.2.0.0 backpropWith     :: Num a     => (forall s. Reifies s W => BVar s a -> BVar s b)     -> a-    -> (b -> b)                 -- ^ Gradient of final result with respect to output of function-    -> (b, a)+    -> (b, (b -> b) -> a) -- ^ Takes function giving gradient of final result given the output of function backpropWith = E.backpropWith E.zfNum {-# INLINE backpropWith #-} @@ -185,18 +192,21 @@     -> a     -> b     -> (c, (a, b))-backprop2 = E.backprop2 E.zfNum E.zfNum E.ofNum+backprop2 f x = second ($ E.ofNum) . E.backprop2 E.zfNum E.zfNum f x {-# INLINE backprop2 #-}  -- | 'Numeric.Backprop.backpropWith2', but with 'Num' constraints instead of -- 'Backprop' constraints.+--+-- Note that argument order changed in v0.2.3.+--+-- @since 0.2.0.0 backpropWith2     :: (Num a, Num b)     => (forall s. Reifies s W => BVar s a -> BVar s b -> BVar s c)     -> a     -> b-    -> (c -> c)                 -- ^ Gradient of final result with respect to output of function-    -> (c, (a, b))+    -> (c, (c -> c) -> (a, b)) -- ^ Takes function giving gradient of final result given the output of function backpropWith2 = E.backpropWith2 E.zfNum E.zfNum {-# INLINE backpropWith2 #-} @@ -342,14 +352,12 @@ -- | 'Numeric.Backprop.collectVar', but with 'Num' constraints instead of -- 'Backprop' constraints. ----- If you are using a list or vector, I recommend using--- <https://hackage.haskell.org/package/vector-sized vector-sized> instead:--- it's a fixed-length vector type with a very appropriate 'Num' instance!+-- Prior to v0.2.3, required a 'Num' constraint on @t a@. collectVar-    :: (Foldable t, Functor t, Num a, Num (t a), Reifies s W)+    :: (Foldable t, Functor t, Num a, Reifies s W)     => t (BVar s a)     -> BVar s (t a)-collectVar = E.collectVar E.afNum E.zfNum E.zfNum+collectVar = E.collectVar E.afNum E.zfNum {-# INLINE collectVar #-}  -- | 'Numeric.Backprop.liftOp', but with 'Num' constraints instead of

@@ -36,9 +36,8 @@ -- with 'BVar's using 'liftOp', 'liftOp1', 'liftOp2', and 'liftOp3'. -- -- If you are writing a library, see--- <https://github.com/mstksg/backprop/wiki/Equipping-your-Library-with-Backprop>--- for a guide for equipping your library with backpropatable operations--- using 'Op's.+-- <https://backprop.jle.im/06-equipping-your-library.html> for a guide for+-- equipping your library with backpropatable operations using 'Op's. --  module Numeric.Backprop.Op (

@@ -33,15 +33,19 @@   , toList   , mapAccumL   , mapAccumR+  , foldr, foldl'   -- * Functor and Applicative   , fmap   , (<$>)   , pure   , liftA2   , liftA3-  -- * Misc+  -- * Numeric   , fromIntegral   , realToFrac+  , round+  , fromIntegral'+  -- * Misc   , E.coerce   ) where @@ -100,19 +104,47 @@ maximum = E.maximum E.addFunc E.zeroFunc {-# INLINE maximum #-} +-- | Lifed 'P.foldr'.  Essentially just 'toList' composed with a normal+-- list 'P.foldr', and is only here for convenience.+--+-- @since 0.2.3.0+foldr+    :: (Traversable t, Backprop a, Reifies s W)+    => (BVar s a -> BVar s b -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldr = E.foldr E.addFunc E.zeroFunc+{-# INLINE foldr #-}++-- | Lifed 'P.foldl''.  Essentially just 'toList' composed with a normal+-- list 'P.foldl'', and is only here for convenience.+--+-- @since 0.2.3.0+foldl'+    :: (Traversable t, Backprop a, Reifies s W)+    => (BVar s b -> BVar s a -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldl' = E.foldl' E.addFunc E.zeroFunc+{-# INLINE foldl' #-}+ -- | Lifted 'P.fmap'.  Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Functor's.+--+-- Prior to v0.2.3, required a 'Backprop' constraint on @f b@. fmap-    :: (Traversable f, Backprop a, Backprop b, Backprop (f b), Reifies s W)+    :: (Traversable f, Backprop a, Backprop b, Reifies s W)     => (BVar s a -> BVar s b)     -> BVar s (f a)     -> BVar s (f b)-fmap = E.fmap E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc+fmap = E.fmap E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE fmap #-}  -- | Alias for 'fmap'. (<$>)-    :: (Traversable f, Backprop a, Backprop b, Backprop (f b), Reifies s W)+    :: (Traversable f, Backprop a, Backprop b, Reifies s W)     => (BVar s a -> BVar s b)     -> BVar s (f a)     -> BVar s (f b)@@ -121,20 +153,24 @@  -- | Lifted 'P.traverse'.  Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Functor's.+--+-- Prior to v0.2.3, required a 'Backprop' constraint on @f (t b)@. traverse-    :: (Traversable t, Applicative f, Foldable f, Backprop a, Backprop b, Backprop (f (t b)), Backprop (t b), Reifies s W)+    :: (Traversable t, Applicative f, Foldable f, Backprop a, Backprop b, Backprop (t b), Reifies s W)     => (BVar s a -> f (BVar s b))     -> BVar s (t a)     -> BVar s (f (t b)) traverse = E.traverse E.addFunc E.addFunc E.addFunc-                      E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc+                      E.zeroFunc E.zeroFunc {-# INLINE traverse #-}  -- | Lifted 'P.liftA2'.  Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Applicative's.+--+-- Prior to v0.2.3, required a 'Backprop' constraint on @f c@. liftA2     :: ( Traversable f, Applicative f-       , Backprop a, Backprop b, Backprop c, Backprop (f c)+       , Backprop a, Backprop b, Backprop c        , Reifies s W        )     => (BVar s a -> BVar s b -> BVar s c)@@ -142,15 +178,17 @@     -> BVar s (f b)     -> BVar s (f c) liftA2 = E.liftA2 E.addFunc E.addFunc E.addFunc-                  E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc+                  E.zeroFunc E.zeroFunc E.zeroFunc {-# INLINE liftA2 #-}  -- | Lifted 'P.liftA3'.  Lifts backpropagatable functions to be -- backpropagatable functions on 'Traversable' 'Applicative's.+--+-- Prior to v0.2.3, required a 'Backprop' constraint on @f d@. liftA3     :: ( Traversable f        , Applicative f-       , Backprop a, Backprop b, Backprop c, Backprop d, Backprop (f d)+       , Backprop a, Backprop b, Backprop c, Backprop d        , Reifies s W        )     => (BVar s a -> BVar s b -> BVar s c -> BVar s d)@@ -159,7 +197,7 @@     -> BVar s (f c)     -> BVar s (f d) liftA3 = E.liftA3 E.addFunc E.addFunc E.addFunc E.addFunc-                  E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc+                  E.zeroFunc E.zeroFunc E.zeroFunc E.zeroFunc {-# INLINE liftA3 #-}  -- | Lifted conversion between two 'P.Integral' instances.@@ -182,6 +220,35 @@ realToFrac = E.realToFrac E.addFunc E.zeroFunc {-# INLINE realToFrac #-} +-- | Lifted version of 'P.round'.+--+-- Gradient should technically diverge whenever the fractional part is 0.5,+-- but does not do this for convenience reasons.+--+-- @since 0.2.3.0+round+    :: (P.RealFrac a, P.Integral b, Reifies s W)+    => BVar s a+    -> BVar s b+round = E.round E.afNum E.zfNum+{-# INLINE round #-}++-- | Lifted version of 'P.fromIntegral', defined to let you return+-- 'P.RealFrac' instances as targets, instead of only other 'P.Integral's.+-- Essentially the opposite of 'round'.+--+-- The gradient should technically diverge whenever the fractional part of+-- the downstream gradient is 0.5, but does not do this for convenience+-- reasons.+--+-- @since 0.2.3.0+fromIntegral'+    :: (P.Integral a, P.RealFrac b, Reifies s W)+    => BVar s a+    -> BVar s b+fromIntegral' = E.fromIntegral' E.afNum E.zfNum+{-# INLINE fromIntegral' #-}+ -- | Lifted version of 'P.toList'.  Takes a 'BVar' of a 'Traversable' of -- items and returns a list of 'BVar's for each item. --@@ -200,25 +267,29 @@  -- | Lifted version of 'P.mapAccumL'. --+-- Prior to v0.2.3, required a 'Backprop' constraint on @t b@.+-- -- @since 0.2.2.0 mapAccumL-    :: (Traversable t, Backprop b, Backprop c, Backprop (t c), Reifies s W)+    :: (Traversable t, Backprop b, Backprop c, Reifies s W)     => (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumL = E.mapAccumL E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc+mapAccumL = E.mapAccumL E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE mapAccumL #-}  -- | Lifted version of 'P.mapAccumR'. --+-- Prior to v0.2.3, required a 'Backprop' constraint on @t b@.+-- -- @since 0.2.2.0 mapAccumR-    :: (Traversable t, Backprop b, Backprop c, Backprop (t c), Reifies s W)+    :: (Traversable t, Backprop b, Backprop c, Reifies s W)     => (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumR = E.mapAccumR E.addFunc E.addFunc E.zeroFunc E.zeroFunc E.zeroFunc+mapAccumR = E.mapAccumR E.addFunc E.addFunc E.zeroFunc E.zeroFunc {-# INLINE mapAccumR #-} 

@@ -27,14 +27,18 @@   , toList   , mapAccumL   , mapAccumR+  , foldr, foldl'   -- * Functor and Applicative   , fmap   , pure   , liftA2   , liftA3-  -- * Misc+  -- * Numeric   , fromIntegral   , realToFrac+  , round+  , fromIntegral'+  -- * Misc   , coerce   ) where @@ -129,21 +133,56 @@         ) {-# INLINE maximum #-} +-- | 'Prelude.Backprop.foldr', but taking explicit 'add' and 'zero'.+--+-- @since 0.2.3.0+foldr+    :: (Traversable t, Reifies s W)+    => AddFunc a+    -> ZeroFunc a+    -> (BVar s a -> BVar s b -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldr af zf f x = P.foldr f x . toList af zf+{-# INLINE foldr #-}++-- | 'Prelude.Backprop.foldl'', but taking explicit 'add' and 'zero'.+--+-- @since 0.2.3.0+foldl'+    :: (Traversable t, Reifies s W)+    => AddFunc a+    -> ZeroFunc a+    -> (BVar s b -> BVar s a -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldl' af zf f x = P.foldl' f x . toList af zf+{-# INLINE foldl' #-}+ -- | 'Prelude.Backprop.fmap', but taking explicit 'add' and 'zero'.+--+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f b)@+-- parameter. fmap     :: (Traversable f, Reifies s W)     => AddFunc a     -> AddFunc b     -> ZeroFunc a     -> ZeroFunc b-    -> ZeroFunc (f b)     -> (BVar s a -> BVar s b)     -> BVar s (f a)     -> BVar s (f b)-fmap afa afb zfa zfb zfbs f = collectVar afb zfb zfbs . P.fmap f . sequenceVar afa zfa+fmap afa afb zfa zfb f = collectVar afb zfb . P.fmap f . sequenceVar afa zfa {-# INLINE fmap #-}  -- | 'Prelude.Backprop.traverse', but taking explicit 'add' and 'zero'.+--+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t b)@+-- and @'ZeroFunc' (f (t b))@ parameters. traverse     :: (Traversable t, Applicative f, Foldable f, Reifies s W)     => AddFunc a@@ -151,19 +190,24 @@     -> AddFunc (t b)     -> ZeroFunc a     -> ZeroFunc b-    -> ZeroFunc (t b)-    -> ZeroFunc (f (t b))     -> (BVar s a -> f (BVar s b))     -> BVar s (t a)     -> BVar s (f (t b))-traverse afa afb aftb zfa zfb zftb zfftb f-        = collectVar aftb zftb zfftb-        . P.fmap (collectVar afb zfb zftb)+traverse afa afb aftb zfa zfb f+        = collectVar aftb zftb+        . P.fmap (collectVar afb zfb)         . P.traverse f         . sequenceVar afa zfa+  where+    zftb = ZF $ P.fmap (runZF zfb)+    {-# INLINE zftb #-} {-# INLINE traverse #-}  -- | 'Prelude.Backprop.liftA2', but taking explicit 'add' and 'zero'.+--+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f c)@+-- parameter. liftA2     :: ( Traversable f        , Applicative f@@ -175,18 +219,21 @@     -> ZeroFunc a     -> ZeroFunc b     -> ZeroFunc c-    -> ZeroFunc (f c)     -> (BVar s a -> BVar s b -> BVar s c)     -> BVar s (f a)     -> BVar s (f b)     -> BVar s (f c)-liftA2 afa afb afc zfa zfb zfc zffc f x y-    = collectVar afc zfc zffc+liftA2 afa afb afc zfa zfb zfc f x y+    = collectVar afc zfc     $ f P.<$> sequenceVar afa zfa x         P.<*> sequenceVar afb zfb y {-# INLINE liftA2 #-}  -- | 'Prelude.Backprop.liftA3', but taking explicit 'add' and 'zero'.+--+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (f d)@+-- parameter. liftA3     :: ( Traversable f        , Applicative f@@ -200,14 +247,13 @@     -> ZeroFunc b     -> ZeroFunc c     -> ZeroFunc d-    -> ZeroFunc (f d)     -> (BVar s a -> BVar s b -> BVar s c -> BVar s d)     -> BVar s (f a)     -> BVar s (f b)     -> BVar s (f c)     -> BVar s (f d)-liftA3 afa afb afc afd zfa zfb zfc zfd zffd f x y z-    = collectVar afd zfd zffd+liftA3 afa afb afc afd zfa zfb zfc zfd f x y z+    = collectVar afd zfd     $ f P.<$> sequenceVar afa zfa x         P.<*> sequenceVar afb zfb y         P.<*> sequenceVar afc zfc z@@ -227,8 +273,7 @@     -> ZeroFunc b     -> BVar s a     -> BVar s b-fromIntegral af zf = liftOp1 af zf . op1 $ \x ->-    (P.fromIntegral x, P.fromIntegral)+fromIntegral af zf = isoVar af zf P.fromIntegral P.fromIntegral {-# INLINE fromIntegral #-}  -- | 'Prelude.Backprop.realToFrac', but taking explicit 'add' and 'zero'.@@ -240,10 +285,34 @@     -> ZeroFunc b     -> BVar s a     -> BVar s b-realToFrac af zf = liftOp1 af zf . op1 $ \x ->-    (P.realToFrac x, P.realToFrac)+realToFrac af zf = isoVar af zf P.realToFrac P.realToFrac {-# INLINE realToFrac #-} +-- | 'Prelude.Backprop.round', but taking explicit 'add' and 'zero'.+--+-- @since 0.2.3.0+round+    :: (P.RealFrac a, P.Integral b, Reifies s W)+    => AddFunc a+    -> ZeroFunc b+    -> BVar s a+    -> BVar s b+round af zf = isoVar af zf P.round P.fromIntegral+{-# INLINE round #-}++-- | 'Prelude.Backprop.fromIntegral'', but taking explicit 'add' and+-- 'zero'.+--+-- @since 0.2.3.0+fromIntegral'+    :: (P.Integral a, P.RealFrac b, Reifies s W)+    => AddFunc a+    -> ZeroFunc b+    -> BVar s a+    -> BVar s b+fromIntegral' af zf = isoVar af zf P.fromIntegral P.round+{-# INLINE fromIntegral' #-}+ -- | 'Prelude.Backprop.length', but taking explicit 'add' and 'zero'. -- -- @since 0.2.2.0@@ -258,6 +327,10 @@  -- | 'Prelude.Backprop.mapAccumL', but taking explicit 'add' and 'zero'. --+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t c)@+-- parameter.+-- -- @since 0.2.2.0 mapAccumL     :: (Traversable t, Reifies s W)@@ -265,19 +338,22 @@     -> AddFunc c     -> ZeroFunc b     -> ZeroFunc c-    -> ZeroFunc (t c)     -> (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumL afb afc zfb zfc zftc f s =-        second (collectVar afc zfc zftc)+mapAccumL afb afc zfb zfc f s =+        second (collectVar afc zfc)       . P.mapAccumL f s       . sequenceVar afb zfb {-# INLINE mapAccumL #-}  -- | 'Prelude.Backprop.mapAccumR', but taking explicit 'add' and 'zero'. --+-- See documentation for 'Numeric.Backprop.Explicitl.collectVar' for+-- information the API change in v0.2.3 that removed the @'ZeroFunc' (t c)@+-- parameter.+-- -- @since 0.2.2.0 mapAccumR     :: (Traversable t, Reifies s W)@@ -285,13 +361,12 @@     -> AddFunc c     -> ZeroFunc b     -> ZeroFunc c-    -> ZeroFunc (t c)     -> (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumR afb afc zfb zfc zftc f s =-        second (collectVar afc zfc zftc)+mapAccumR afb afc zfb zfc f s =+        second (collectVar afc zfc)       . P.mapAccumR f s       . sequenceVar afb zfb {-# INLINE mapAccumR #-}

@@ -26,15 +26,19 @@   , toList   , mapAccumL   , mapAccumR+  , foldr, foldl'   -- * Functor and Applicative   , fmap   , (<$>)   , pure   , liftA2   , liftA3-  -- * Misc+  -- * Numeric   , fromIntegral   , realToFrac+  , round+  , fromIntegral'+  -- * Misc   , E.coerce   ) where @@ -97,19 +101,47 @@ maximum = E.maximum E.afNum E.zfNum {-# INLINE maximum #-} +-- | 'Prelude.Backprop.foldr', but with 'Num' constraints instead of+-- 'Backprop' constraints.+--+-- @since 0.2.3.0+foldr+    :: (Traversable t, Num a, Reifies s W)+    => (BVar s a -> BVar s b -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldr = E.foldr E.afNum E.zfNum+{-# INLINE foldr #-}++-- | 'Prelude.Backprop.foldl'', but with 'Num' constraints instead of+-- 'Backprop' constraints.+--+-- @since 0.2.3.0+foldl'+    :: (Traversable t, Num a, Reifies s W)+    => (BVar s b -> BVar s a -> BVar s b)+    -> BVar s b+    -> BVar s (t a)+    -> BVar s b+foldl' = E.foldl' E.afNum E.zfNum+{-# INLINE foldl' #-}+ -- | 'Prelude.Backprop.fmap', but with 'Num' constraints instead of -- 'Backprop' constraints.+--+-- Prior to v0.2.3, required a 'Num' constraint on @f b@. fmap-    :: (Traversable f, Num a, Num b, Num (f b), Reifies s W)+    :: (Traversable f, Num a, Num b, Reifies s W)     => (BVar s a -> BVar s b)     -> BVar s (f a)     -> BVar s (f b)-fmap = E.fmap E.afNum E.afNum E.zfNum E.zfNum E.zfNum+fmap = E.fmap E.afNum E.afNum E.zfNum E.zfNum {-# INLINE fmap #-}  -- | Alias for 'fmap'. (<$>)-    :: (Traversable f, Num a, Num b, Num (f b), Reifies s W)+    :: (Traversable f, Num a, Num b, Reifies s W)     => (BVar s a -> BVar s b)     -> BVar s (f a)     -> BVar s (f b)@@ -118,35 +150,44 @@  -- | 'Prelude.Backprop.traverse', but with 'Num' constraints instead of -- 'Backprop' constraints.+--+-- See <https://hackage.haskell.org/package/vector-sized vector-sized> for+-- a fixed-length vector type with a very appropriate 'Num' instance!+--+-- Prior to v0.2.3, required a 'Num' constraint on @f (t b)@. traverse-    :: (Traversable t, Applicative f, Foldable f, Num a, Num b, Num (f (t b)), Num (t b), Reifies s W)+    :: (Traversable t, Applicative f, Foldable f, Num a, Num b, Num (t b), Reifies s W)     => (BVar s a -> f (BVar s b))     -> BVar s (t a)     -> BVar s (f (t b))-traverse = E.traverse E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum E.zfNum+traverse = E.traverse E.afNum E.afNum E.afNum E.zfNum E.zfNum {-# INLINE traverse #-}  -- | 'Prelude.Backprop.liftA2', but with 'Num' constraints instead of -- 'Backprop' constraints.+--+-- Prior to v0.2.3, required a 'Num' constraint on @f c@. liftA2     :: ( Traversable f        , Applicative f-       , Num a, Num b, Num c, Num (f c)+       , Num a, Num b, Num c        , Reifies s W        )     => (BVar s a -> BVar s b -> BVar s c)     -> BVar s (f a)     -> BVar s (f b)     -> BVar s (f c)-liftA2 = E.liftA2 E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum E.zfNum+liftA2 = E.liftA2 E.afNum E.afNum E.afNum E.zfNum E.zfNum E.zfNum {-# INLINE liftA2 #-}  -- | 'Prelude.Backprop.liftA3', but with 'Num' constraints instead of -- 'Backprop' constraints.+--+-- Prior to v0.2.3, required a 'Num' constraint on @f d@. liftA3     :: ( Traversable f        , Applicative f-       , Num a, Num b, Num c, Num d, Num (f d)+       , Num a, Num b, Num c, Num d        , Reifies s W        )     => (BVar s a -> BVar s b -> BVar s c -> BVar s d)@@ -155,7 +196,7 @@     -> BVar s (f c)     -> BVar s (f d) liftA3 = E.liftA3 E.afNum E.afNum E.afNum E.afNum-                  E.zfNum E.zfNum E.zfNum E.zfNum E.zfNum+                  E.zfNum E.zfNum E.zfNum E.zfNum {-# INLINE liftA3 #-}  -- | 'Prelude.Backprop.fromIntegral', but with 'Num' constraints instead of@@ -180,6 +221,28 @@ realToFrac = E.realToFrac E.afNum E.zfNum {-# INLINE realToFrac #-} +-- | 'Prelude.Backprop.round', but with 'Num' constraints instead of+-- 'Backprop' constraints.+--+-- @since 0.2.3.0+round+    :: (P.RealFrac a, P.Integral b, Reifies s W)+    => BVar s a+    -> BVar s b+round = E.round E.afNum E.zfNum+{-# INLINE round #-}++-- | 'Prelude.Backprop.fromIntegral'', but with 'Num' constraints instead+-- of 'Backprop' constraints.+--+-- @since 0.2.3.0+fromIntegral'+    :: (P.Integral a, P.RealFrac b, Reifies s W)+    => BVar s a+    -> BVar s b+fromIntegral' = E.fromIntegral' E.afNum E.zfNum+{-# INLINE fromIntegral' #-}+ -- | 'Prelude.Backprop.toList', but with 'Num' constraints instead of -- 'Backprop' constraints. --@@ -194,26 +257,30 @@ -- | 'Prelude.Backprop.mapAccumL', but with 'Num' constraints instead of -- 'Backprop' constraints. --+-- Prior to v0.2.3, required a 'Num' constraint on @t b@.+-- -- @since 0.2.2.0 mapAccumL-    :: (Traversable t, Num b, Num c, Num (t c), Reifies s W)+    :: (Traversable t, Num b, Num c, Reifies s W)     => (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumL = E.mapAccumL E.afNum E.afNum E.zfNum E.zfNum E.zfNum+mapAccumL = E.mapAccumL E.afNum E.afNum E.zfNum E.zfNum {-# INLINE mapAccumL #-}  -- | 'Prelude.Backprop.mapAccumR', but with 'Num' constraints instead of -- 'Backprop' constraints. --+-- Prior to v0.2.3, required a 'Num' constraint on @t b@.+-- -- @since 0.2.2.0 mapAccumR-    :: (Traversable t, Num b, Num c, Num (t c), Reifies s W)+    :: (Traversable t, Num b, Num c, Reifies s W)     => (BVar s a -> BVar s b -> (BVar s a, BVar s c))     -> BVar s a     -> BVar s (t b)     -> (BVar s a, BVar s (t c))-mapAccumR = E.mapAccumR E.afNum E.afNum E.zfNum E.zfNum E.zfNum+mapAccumR = E.mapAccumR E.afNum E.afNum E.zfNum E.zfNum {-# INLINE mapAccumR #-}