path: root/notes/43_aaai_accept_bar_experiments.md

# AAAI Accept-Bar Experiments

Status: **complete, paper-grade local outputs** (2026-07-21).  This note
records the experiments added after the focused-paper review.  It supplements
the pre-existing master reference (note 42); it does not rewrite the paper.

## 1. What was added

Four missing controls are now implemented and run:

1. FA and DFA initialization-cost calibration at hidden depths `{1,2,3,4,6}`.
2. FA finite-time gap and operator-predictor errors over depth
   `{1,2,3,4,6}` and width `{32,64,96}`.
3. Same-rate, exact-initialization-rate-compensated, and independently tuned
   BP/FA learning-rate controls.
4. A non-MLP initialization validation: a two-convolution-layer CNN with
   cross-entropy loss on an MNIST subset, for both FA and DFA.

Two three-panel composite figures are generated from these results plus the
existing random-direction-deletion negative control.

## 2. Shared implementation checks

New shared implementation: `scripts/feedback_rules.py`.

- Arbitrary-depth manual BP agrees with autograd to `5.55e-17` max error.
- FA and DFA output-layer gradients agree with BP exactly.
- A 4096-feedback-draw numerical moment test for each rule lands within the
  Monte Carlo interval predicted by the exact initialization result.
- The CNN manual BP agrees with autograd to `5.55e-17`; FA and DFA preserve
  the exact output gradient.

Commands:

```bash
python scripts/test_feedback_rules.py
python scripts/cnn_initialization_validation.py --self-test
```

## 3. FA/DFA initialization calibration across depth

Configuration:

- synthetic random regression, `N=64`, input dimension 16, output dimension 4;
- fixed width 64;
- hidden depth `{1,2,3,4,6}`;
- 6 fixed forward initializations per depth;
- 256 independent feedback draws per forward initialization and rule;
- real FA/DFA backward pass for every draw (15,360 feedback evaluations).

The predicted quantity is the relative deficit in BP's first-order loss
decrease:

```text
1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2.
```

Depth-aggregated results:

| depth | predicted | FA measured | DFA measured |
|---:|---:|---:|---:|
| 1 | 0.050380 | 0.050200 | 0.050578 |
| 2 | 0.276966 | 0.275983 | 0.278078 |
| 3 | 0.402635 | 0.402218 | 0.402282 |
| 4 | 0.458542 | 0.457015 | 0.456779 |
| 6 | 0.623193 | 0.624234 | 0.623791 |

Across all 60 rule/depth/init rows, max absolute calibration error is
`0.009164`; max standardized error is `2.412` Monte Carlo standard errors.
The same exact prediction covers FA and DFA even though their feedback paths
differ.

Output: `outputs/aaai_depth_experiments/initialization_rows.csv` and
`initial_deficit_calibration.png`.

## 4. Finite-time FA cost across depth and width

Configuration:

- same synthetic task family, `N=64`, output dimension 4;
- hidden depth `{1,2,3,4,6}`, width `{32,64,96}`;
- 4 forward initializations x 8 FA feedback initializations per cell;
- full-batch SGD, fixed learning rate `1e-3`, horizon `T=50`;
- early operator snapshot `s=20`;
- 480 FA trajectories plus cached matched BP trajectories.

Curve uncertainty is computed across forward-initialization means (feedback
draws are nested within each initialization), rather than treating all 32
feedback trajectories in a cell as independent.

At width 64, the measured mean gap rises with depth:

| depth | measured | frozen `K(0)` | linear velocity | early retangent |
|---:|---:|---:|---:|---:|
| 1 | 0.042081 | 0.046148 | 0.041747 | 0.043214 |
| 2 | 0.152697 | 0.176001 | 0.150869 | 0.159079 |
| 3 | 0.252548 | 0.301208 | 0.246641 | 0.263831 |
| 4 | 0.281210 | 0.360586 | 0.274093 | 0.302059 |
| 6 | 0.346053 | 0.518519 | 0.312131 | 0.371649 |

For the same fixed-width architecture, the independent per-block alignment
proxy `prod_l 1/D_l` falls from `3.91e-3` at depth 1 to `3.39e-21` at depth 6,
an 18.06-order-of-magnitude change.  Over that sweep, the measured initial
deficit changes from `0.0504` to `0.6240`, and the measured finite-time gap
from `0.0421` to `0.3461`.  Thus depth makes the matrix-level mismatch
astronomically worse without producing a commensurate optimization-cost
explosion.  The direct comparison is saved in
`mismatch_vs_cost_by_depth.{csv,png}`.

Across the full 480-row grid:

| predictor | MAE | relative MAE vs mean measured gap | correlation |
|---|---:|---:|---:|
| frozen `K(0)` | 0.06693 | 28.6% | 0.97166 |
| linear operator velocity | 0.01043 | 4.45% | 0.99492 |
| early retangent | 0.01394 | 5.95% | 0.99830 |

The frozen operator increasingly underpredicts progress at larger depth.  Two
snapshots recover most of the magnitude, supporting the scoped statement that
the later cost depends on operator evolution.  The finite-time result remains
FA-only.

Output: `outputs/aaai_depth_experiments/finite_time_rows.csv`,
`finite_time_metrics.csv`, and the corresponding plots.

### Parameter-matched supplement

The fixed-width result is accompanied by an equal-parameter-count control.
Using depth-2 width-64 as the target (`P=5376`, no biases), the closest
uniform widths are `{269,64,47,39,31}` for depths `{1,2,3,4,6}`, with all
parameter counts within 49 of the target.  Each cell contains 4 forward
initializations x 8 feedback initializations; error bars again use the four
forward-initialization means.

| depth | width | parameters | measured gap |
|---:|---:|---:|---:|
| 1 | 269 | 5380 | 0.02695 |
| 2 | 64 | 5376 | 0.15270 |
| 3 | 47 | 5358 | 0.18857 |
| 4 | 39 | 5343 | 0.37599 |
| 6 | 31 | 5425 | 0.21024 |

The gap is not monotone in depth once parameter count is fixed: it peaks at
depth 4 and falls at depth 6.  This is another reason not to infer optimization
cost directly from the number of independently mismatched matrices.  Output:
`outputs/parameter_matched_depth_control/`.

## 5. Learning-rate alternative explanation

Configuration:

- fixed width 64, depth `{1,2,3,4,6}`, `N=128`, 200 steps;
- baseline BP/FA learning rate `1e-3`;
- compensated FA rate `1e-3 / rho`, with `rho` read from one BP backward pass;
- independent BP and FA rates selected using 4 tuning initializations x 4 FA
  feedback seeds from
  `{0.00025,0.0005,0.001,0.002,0.004,0.008,0.016,0.032,0.064,0.128,0.256}`;
- stability-confirmed evaluation on 6 new forward initializations x 8 feedback
  initializations; all 720 reported evaluation rows stable.

Learning-rate curve uncertainty is likewise computed across the six forward
initializations, with eight feedback draws nested within each one.

| depth | same-rate gap | compensated gap | reduction | independently tuned gap |
|---:|---:|---:|---:|---:|
| 1 | 0.05906 | 0.04053 | 31.4% | 0.56940 |
| 2 | 0.19916 | 0.09729 | 51.2% | 0.64210 |
| 3 | 0.27346 | 0.16263 | 40.5% | 1.13002 |
| 4 | 0.29340 | 0.18442 | 37.1% | 1.30812 |
| 6 | 0.40664 | 0.28850 | 29.1% | 1.75739 |

Initialization compensation removes a material part of the same-rate gap but
does not remove it at any tested depth.  Independent tuning does not rescue
FA: the selected BP rates are `0.064-0.128`, whereas the selected FA rates
fall from `0.064` at depth 1 to stability-confirmed `0.004` at depth 6.  BP therefore makes much
more progress in the same 200-step budget.  This result should be described as
a fixed-horizon optimization comparison, not a claim about asymptotic
convergence.

Output: `outputs/learning_rate_compensation/`.

## 6. Cross-entropy CNN initialization validation

Configuration:

- MNIST subset, 128 examples resized to `8x8`;
- `Conv(1,8,3x3) -> ReLU -> Conv(8,12,3x3) -> ReLU -> global average pool -> linear`;
- cross-entropy loss;
- 4 forward initializations;
- 2048 real-backward feedback draws per initialization and rule (FA and DFA).

The predicted deficit spans `0.592-0.816`.  Across the eight rows, max
absolute calibration error is `0.008309`, and max standardized error is only
`1.308` Monte Carlo standard errors.  This validates the initialization
identity outside squared-loss MLPs without making a finite-time CNN claim.

Output: `outputs/cnn_initialization_validation/`.

## 7. Figures

- `outputs/aaai_main_figures/figure1_mismatch_vs_initial_cost.{png,pdf}`:
  BP/FA/DFA schematic; minimax mismatch plus the 250x null-model
  overprediction; joint initialization calibration across rule, depth, MNIST
  MLP, and cross-entropy CNN.
- `outputs/aaai_main_figures/figure2_training_dynamics.{png,pdf}`:
  measured finite-time cost over depth/width; frozen-operator calibration;
  two-snapshot calibration.
- Supplemental learning-rate figures:
  `outputs/learning_rate_compensation/learning_rate_control_by_depth.png` and
  `learning_rate_tuning_curves.png`.
- Direct depth contrast:
  `outputs/aaai_depth_experiments/mismatch_vs_cost_by_depth.{csv,png}`.
- Parameter-matched supplement:
  `outputs/parameter_matched_depth_control/parameter_matched_gap_by_depth.png`.

## 8. Claim boundary after these experiments

Supported:

- The exact expected initialization result applies to both FA and DFA under
  independent zero-mean feedback, and is validated through depth 6.
- It also holds for a cross-entropy CNN in the tested convolutional feedback
  implementation.
- Simple initial-rate compensation reduces but does not eliminate the
  finite-time FA/BP gap in the tested MLP setting.
- FA's later gap and frozen-model error grow with depth; two early operator
  snapshots remain predictive.

Not supported / not claimed:

- DFA finite-time dynamics.
- CNN finite-time training equivalence or modern large-scale scaling.
- A universal absence of width/depth thresholds.
- A per-eigenmode erosion theorem.

## 9. Reproduction commands

```bash
python scripts/aaai_depth_experiments.py --part init \
  --depths 1 2 3 4 6 --width 64 --train-samples 64 \
  --init-seeds 6 --feedback-draws 256 --torch-threads 16 \
  --outdir outputs/aaai_depth_experiments

python scripts/aaai_depth_experiments.py --part finite \
  --depths 1 2 3 4 6 --finite-widths 32 64 96 --train-samples 64 \
  --finite-init-seeds 4 --finite-feedback-seeds 8 \
  --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 \
  --outdir outputs/aaai_depth_experiments

python scripts/aaai_depth_experiments.py --part summarize \
  --outdir outputs/aaai_depth_experiments

python scripts/parameter_matched_depth_control.py \
  --depths 1 2 3 4 6 --reference-depth 2 --reference-width 64 \
  --train-samples 64 --init-seeds 4 --feedback-seeds 8 \
  --lr 0.001 --horizon 50 --early-step 20 --torch-threads 16 \
  --outdir outputs/parameter_matched_depth_control

python scripts/learning_rate_compensation.py \
  --depths 1 2 3 4 6 --width 64 --train-samples 128 --steps 200 \
  --base-lr 0.001 --tune-init-seeds 4 --tune-feedback-seeds 4 \
  --eval-init-seeds 6 --eval-feedback-seeds 8 --torch-threads 16 \
  --outdir outputs/learning_rate_compensation

python scripts/cnn_initialization_validation.py \
  --train-samples 128 --image-size 8 --channels 8 12 \
  --init-seeds 4 --feedback-draws 2048 --torch-threads 8 \
  --outdir outputs/cnn_initialization_validation

python scripts/plot_aaai_main_figures.py \
  --outdir outputs/aaai_main_figures
```