diff options
Diffstat (limited to 'THEORY.md')
| -rw-r--r-- | THEORY.md | 1 |
1 files changed, 1 insertions, 0 deletions
@@ -356,6 +356,7 @@ work `S_l=sum_{j>l} F_j`. |:--|:--|:--|:--|:--|:--| | BP | forward + reverse | exact transposed Jacobians | ordinary supervised loss | about one reverse pass | saved activations / graph | | FA | forward + reverse-like serial feedback | fixed random feedback | ordinary loss | `O(F)` feedback work | activations + feedback states | +| convolutional hierarchical FA | forward + residual-DAG feedback | independent random 3x3 feedback tensors; exact parameter-free shortcut graph | ordinary loss | one feedback convolution per non-stem forward convolution (`0.988x` forward MACs at ResNet-20, plus local correlations) | hidden feedback maps + forward caches | | DFA | forward + direct feedback | fixed random output maps | ordinary loss | `O(sum_l d_l d_out)` | activations + feedback states | | learned NP, simultaneous | ordinary forward; one duplicate clean plus `2K` noisy full forwards every `e` batches | learned direct maps; no weight transport | `2KB/e` per ordinary batch | `(1+2K)F/e` | up to `2KB` noisy examples in the vectorized implementation | | learned NP, layerwise | ordinary forward; clean baseline plus antithetic suffix replays | learned direct maps | `(1+2KH)B/e` | `(F + 2K sum_l S_l)/e`, quadratic in depth for balanced layers | serial suffix replay, no `2K` batch expansion | |