1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
|
# Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians
This repository contains the implementation code for manuscript: <br>
__Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians__ <br>
## Overview
In this work we show that RNN training can be improved by pushing Lyapunov exponents of the forward pass to zero during or before training.
## Installation
#### Prerequisites
- Download [Julia](https://julialang.org/downloads/)
#### Dependencies
- Julia (1.6)
- Flux, BackwardsLinalg
## Getting started
To install the required packages, run the following in the julia REPL after installing Julia:
```
using Pkg
for pkg in ["Flux", "BackwardsLinalg"]
Pkg.add(pkg)
end
```
For example, to train an RNN on tracking delayed XOR task, run:
```
include("GradientFlossing_ExampleCode.jl")
# setting parameters:
N, E, Ef, Ei, Ep, Ni, B, S, T, Tp, Ti, sIC, sIn, sNet, sONS, lr, b1, b2, IC, g, gbar, I1, delay, wsS, wsM, wrS, wrM, bS, bM, nLE, task, intype, Lwnt=
80, 3001, 100, 500, 500, 2, 16, 1, 300, 55, 300, 1,1,1,1, 0.001f0, 0.9, 0.999, 1, 1.0, 0.0, 1.0,10, 1.0f0, 0.0f0, 1.0f0, 0.0f0, 0.1f0, 0.0f0,75, -1, 3, 0.0
trainRNNflossing(N, E, Ef, Ei, Ep, Ni, B, S, T, Tp, Ti, sIC, sIn, sNet, sONS, lr, b1, b2, IC, g, gbar, I1, delay, wsS, wsM, wrS, wrM, bS, bM, nLE, task, intype, Lwnt)
```
## Repository Overview
_GradientFlossing_ExampleCode.jl_:\
Example scripts for training networks with gradient flossing before training, with gradient flossing before and during training and without gradient flossing.
_GradientFlossing_XOR.jl_:\
Generates input and target output for copy task and delayed XOR task.
<!---
runOneStimulus.jl trains an RNN on tracking one OU-signal showing that the network becomes more tightly balanced over training epochs.\
runTwoStimuli.jl trains an RNN on two OU-signal stimulus showing that the network becomes more tightly balanced over training epochs and breaks up into two weakly-connected subnetworks.\
runTheeStimuli.jl trains an RNN on two OU-signal stimulus showing that the network becomes more tightly balanced over training epochs and breaks up into three weakly-connected subnetworks.\
-->
<!---
### Training dynamics of eigenvalues:
Here is a visualization of the recurrent weight matrix and the eigenvalues throughout across training epochs.
-->
### Implementation details
A full specification of packages used and their versions can be found in _packages.txt_ .\
For learning rates the default ADAM parameters were used to avoid any impression of fine-tuning.\
All simulations were run on a single CPU and took on the order of minutes to a few of hours.
<!---
### figures/
Contains all figures of the main text and the supplement.
-->
<!---
### tex/
Contains the raw text of the main text and the supplement.
-->
|
