# github.com-HuwCampbell-grenade_-_2017-05-24_22-40-05

## Audio Preview

# Share or Embed This Item

# Flag this item for

- Publication date
- 2017-05-24

`HuwCampbell-grenade_-_2017-05-24_22-40-05.bundle`and run:

` git clone HuwCampbell-grenade_-_2017-05-24_22-40-05.bundle -b master `

Practical Deep Learning in Haskell

# Grenade

`First shalt thou take out the Holy Pin, then shalt thou count to three, no more, no less.Three shall be the number thou shalt count, and the number of the counting shall be three.Four shalt thou not count, neither count thou two, excepting that thou then proceed to three.Five is right out.`

💣 Machine learning which might blow up in your face 💣

Grenade is a composable, dependently typed, practical, and fast recurrent neural network libraryfor concise and precise specifications of complex networks in Haskell.

As an example, a network which can achieve ~1.5% error on MNIST can bespecified and initialised with random weights in a few lines of code with```haskelltype MNIST = Network '[ Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu , Convolution 10 16 5 5 1 1, Pooling 2 2 2 2, FlattenLayer, Relu , FullyConnected 256 80, Logit, FullyConnected 80 10, Logit] '[ 'D2 28 28, 'D3 24 24 10, 'D3 12 12 10, 'D3 12 12 10 , 'D3 8 8 16, 'D3 4 4 16, 'D1 256, 'D1 256 , 'D1 80, 'D1 80, 'D1 10, 'D1 10]

randomMnist :: MonadRandom m => m MNISTrandomMnist = randomNetwork```

And that's it. Because the types are so rich, there's no specific term level coderequired to construct this network; although it is of course possible andeasy to construct and deconstruct the networks and layers explicitly oneself.

If recurrent neural networks are more your style, you can try defining something"unreasonably effective"with`haskelltype Shakespeare = RecurrentNetwork '[ R (LSTM 40 80), R (LSTM 80 40), F (FullyConnected 40 40), F Logit] '[ 'D1 40, 'D1 80, 'D1 40, 'D1 40, 'D1 40 ]`

## Design

Networks in Grenade can be thought of as a heterogeneous lists of layers, wheretheir type includes not only the layers of the network, but also the shapes ofdata that are passed between the layers.

The definition of a network is surprisingly simple:```haskelldata Network :: [*] -> [Shape] -> * where NNil :: SingI i => Network '[] '[i]

`(:~>) :: (SingI i, SingI h, Layer x i h) => !x -> !(Network xs (h ': hs)) -> Network (x ': xs) (i ': h ': hs)`

```

The `Layer x i o`

constraint ensures that the layer `x`

can sensibly perform atransformation between the input and output shapes `i`

and `o`

.

The lifted data kind `Shape`

defines our 1, 2, and 3 dimension types, used todeclare what shape of data is passed between the layers.

In the MNIST example above, the input layer can be seen to be a two dimensional(`D2`

), image with 28 by 28 pixels. When the first *Convolution* layer runs, itoutputs a three dimensional (`D3`

) 24x24x10 image. The last item in the list isone dimensional (`D1`

) with 10 values, representing the categories of the MNISTdata.

## Usage

To perform back propagation, one can call the eponymous function`haskellbackPropagate :: forall shapes layers. Network layers shapes -> S (Head shapes) -> S (Last shapes) -> Gradients layers`

which takes a network, appropriate input and target data, and returns theback propagated gradients for the network. The shapes of the gradients areappropriate for each layer, and may be trivial for layers like `Relu`

whichhave no learnable parameters.

The gradients however can always be applied, yielding a new (hopefully better)layer with`haskellapplyUpdate :: LearningParameters -> Network ls ss -> Gradients ls -> Network ls ss`

Layers in Grenade are represented as Haskell classes, so creating one's own iseasy in downstream code. If the shapes of a network are not specified correctlyand a layer can not sensibly perform the operation between two shapes, thenit will result in a compile time error.

## Composition

Networks and Layers in Grenade are easily composed at the type level. As a `Network`

is an instance of `Layer`

, one can use a trained Network as a small component in alarger network easily. Furthermore, we provide 2 layers which are designed to runlayers in parallel and merge their output (either by concatenating them across onedimension or summing by pointwise adding their activations). This allows one towrite any Network which can be expressed as aseries parallel graph.

A residual network layer specification for instance could be written as`haskelltype Residual net = Merge Trivial net`

If the type `net`

is an instance of `Layer`

, then `Residual net`

will be too. It willrun the network, while retaining its input by passing it through the `Trivial`

layer,and merge the original image with the output.

See the MNISTexample, which has been overengineered to contain both residual style learning as wellas inception style convolutions.

## Generative Adversarial Networks

As Grenade is purely functional, one can compose its training functions in flexibleways. GAN-MNISTexample displays an interesting, type safe way of writing a generative adversarialtraining function in 10 lines of code.

## Layer Zoo

Grenade layers are normal haskell data types which are an instance of `Layer`

, soit's easy to build one's own downstream code. We do however provide a decent setof layers, including convolution, deconvolution, pooling, pad, crop, logit, relu,elu, tanh, and fully connected.

## Build Instructions

Grenade is most easily built with the mafiascript that is located in the repository. You will also need the `lapack`

and`blas`

libraries and development tools. Once you have all that, Grenade can bebuild using:

`./mafia build`

and the tests run using:

`./mafia test`

Grenade builds with ghc 7.10 and 8.0.

## Thanks

Writing a library like this has been on my mind for a while now, but a big shoutout must go to Justin Le, whosedependently typed fully connected networkinspired me to get cracking, gave many ideas for the type level tools Ineeded, and was a great starting point for writing this library.

## Performance

Grenade is backed by hmatrix, BLAS, and LAPACK, with critical functions optimisedin C. Using the im2col trick popularised by Caffe, it should be sufficient formany problems.

Being purely functional, it should also be easy to run batches in parallel, whichwould be appropriate for larger networks, my current examples however are singlethreaded.

Training 15 generations over Kaggle's 41000 sample MNIST training set on a singlecore took around 12 minutes, achieving 1.5% error rate on a 1000 sample holdout set.

## Contributing

Contributions are welcome.

Source: https://github.com/HuwCampbell/grenade

Uploader: HuwCampbell

Upload date: 2017-05-24

- Addeddate
- 2017-05-24 23:05:43

- Identifier
- github.com-HuwCampbell-grenade_-_2017-05-24_22-40-05

- Originalurl
- https://github.com/HuwCampbell/grenade

- Pushed_date
- 2017-05-24 22:40:05

- Scanner
- Internet Archive Python library 1.5.0

- Uploaded_with
- iagitup - v1.0

- Year
- 2017