Test first order FA tangent hierarchy

1 files changed, 222 insertions, 0 deletions

diff --git a/notes/31_first_order_tangent_hierarchy_try.md b/notes/31_first_order_tangent_hierarchy_try.md
new file mode 100644
index 0000000..663ae35
--- /dev/null
+++ b/notes/31_first_order_tangent_hierarchy_try.md
@@ -0,0 +1,222 @@
+# First-Order FA Tangent-Hierarchy Try
+
+This note records the first direct attempt to use tangent-hierarchy ideas for
+finite-time FA/BP gap prediction.
+
+## Goal
+
+Test whether finite-time gap can be improved by replacing the fixed initial
+operator:
+
+```text
+K_t = K_0
+```
+
+with a first-order operator trajectory:
+
+```text
+K_t = K_0 + t V_0
+```
+
+where `V_0` is either:
+
+```text
+early average velocity: (K_s - K_0) / s
+```
+
+or:
+
+```text
+infinitesimal derivative: (K_epsilon - K_0) / epsilon
+```
+
+This is the smallest test of the FA tangent-hierarchy idea.
+
+## Experiment 1: Early Average Velocity
+
+Script:
+
+```text
+scripts/compressed_operator_predictor.py
+```
+
+Run:
+
+```text
+python scripts/compressed_operator_predictor.py \
+  --width 64 \
+  --hidden-layers 2 \
+  --train-samples 128 \
+  --target-steps 50 \
+  --early-steps 1 2 5 \
+  --init-seeds 3 \
+  --feedback-seeds 6 \
+  --lr 1e-3 \
+  --torch-threads 8 \
+  --outdir outputs/fa_tangent_hierarchy_first_order_try
+```
+
+Plots:
+
+```text
+outputs/fa_tangent_hierarchy_first_order_try/first_order_prediction_scatter_by_early_step.png
+outputs/fa_tangent_hierarchy_first_order_try/first_order_error_metrics_by_early_step.png
+outputs/fa_tangent_hierarchy_first_order_try/first_order_gap_distribution_overlay.png
+```
+
+Summary:
+
+| early s | predictor | MAE | bias | corr |
+|---:|---|---:|---:|---:|
+| 1 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
+| 1 | linear velocity | 0.022925 | -0.020753 | 0.742632 |
+| 2 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
+| 2 | linear velocity | 0.012290 | -0.008418 | 0.815872 |
+| 5 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
+| 5 | linear velocity | 0.012232 | -0.012232 | 0.960370 |
+| 5 | retangent | 0.016770 | +0.016770 | 0.971053 |
+
+Interpretation:
+
+```text
+early average velocity improves magnitude substantially
+```
+
+At `s=2` or `s=5`, MAE is nearly cut in half relative to fixed `K0`.
+
+But:
+
+```text
+linear extrapolation shifts from positive bias to negative bias
+```
+
+So the first-order average velocity captures the direction of operator drift,
+but extrapolating it linearly to `T=50` overcorrects.
+
+This is useful: the tangent-hierarchy route is real, but the model should not
+be a naive global linear extrapolation.
+
+## Experiment 2: Infinitesimal Derivative Probe
+
+Script:
+
+```text
+scripts/fa_tangent_hierarchy_derivative_probe.py
+```
+
+Run:
+
+```text
+python scripts/fa_tangent_hierarchy_derivative_probe.py \
+  --width 64 \
+  --hidden-layers 2 \
+  --train-samples 128 \
+  --target-steps 50 \
+  --epsilon 0.05 \
+  --init-seeds 3 \
+  --feedback-seeds 6 \
+  --lr 1e-3 \
+  --torch-threads 8 \
+  --outdir outputs/fa_tangent_hierarchy_derivative_probe_eps005
+```
+
+Result:
+
+```text
+fixed MAE      = 0.0237897
+fixed bias     = +0.0237897
+derivative MAE = 270.362
+derivative bias= -270.212
+```
+
+The infinitesimal first derivative is unusable when extrapolated linearly to
+`T=50`.
+
+Likely reasons:
+
+```text
+1. K_t is not globally linear for 50 steps;
+2. the first derivative creates an indefinite operator path that can become
+   unstable when extrapolated;
+3. ReLU gates and FA alignment dynamics make the actual early average velocity
+   very different from the infinitesimal derivative;
+4. higher-order hierarchy terms are not small over this horizon.
+```
+
+This is not a normalization bug. The same setup shows fixed `K0` is good at
+very short horizons and that early average velocity improves finite-time
+magnitude.
+
+## Consequence
+
+The simple theorem:
+
+```text
+K_t = K_0 + t dot_K_0
+```
+
+is not enough for practical `T=50` prediction.
+
+The better finite-time object is:
+
+```text
+K_t = K_0 + A_t
+```
+
+where `A_t` should be a bounded short-time alignment-gain path, not an
+unbounded linear extrapolation.
+
+Empirically, the useful object is:
+
+```text
+average early velocity over s=2 or s=5
+```
+
+not the infinitesimal `dot_K_0`.
+
+## Revised Theory Direction
+
+We should pursue one of these:
+
+1. **Integrated short-time hierarchy**
+
+   Predict:
+
+   ```text
+   K_s - K_0 = integral_0^s dot_K_u du
+   ```
+
+   rather than only `dot_K_0`.
+
+2. **Stable alignment-gain scalar**
+
+   Decompose:
+
+   ```text
+   speed_FA,t = output_speed_t + hidden_alignment_gain_t
+   ```
+
+   and model the hidden alignment gain as a bounded increasing process.
+
+3. **Piecewise retangent estimator**
+
+   For empirical validation, use:
+
+   ```text
+   measured K_s
+   ```
+
+   as a conditional estimator, and do not claim it is architecture-only.
+
+The most paper-safe claim:
+
+```text
+The exact e_0 theorem gives the initial operator cost. For finite time, the
+successful predictor is an early-time integrated tangent-operator estimator,
+which is an empirical finite-difference version of a truncated FA tangent
+hierarchy. A pure infinitesimal first-derivative extrapolation is not stable
+over nonlocal horizons.
+```
+
+This result is useful because it prevents us from overclaiming a too-simple
+first-order theory.