blob: 663ae3530b9ee1b5de86c91f446780ed03355230 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
|
# First-Order FA Tangent-Hierarchy Try
This note records the first direct attempt to use tangent-hierarchy ideas for
finite-time FA/BP gap prediction.
## Goal
Test whether finite-time gap can be improved by replacing the fixed initial
operator:
```text
K_t = K_0
```
with a first-order operator trajectory:
```text
K_t = K_0 + t V_0
```
where `V_0` is either:
```text
early average velocity: (K_s - K_0) / s
```
or:
```text
infinitesimal derivative: (K_epsilon - K_0) / epsilon
```
This is the smallest test of the FA tangent-hierarchy idea.
## Experiment 1: Early Average Velocity
Script:
```text
scripts/compressed_operator_predictor.py
```
Run:
```text
python scripts/compressed_operator_predictor.py \
--width 64 \
--hidden-layers 2 \
--train-samples 128 \
--target-steps 50 \
--early-steps 1 2 5 \
--init-seeds 3 \
--feedback-seeds 6 \
--lr 1e-3 \
--torch-threads 8 \
--outdir outputs/fa_tangent_hierarchy_first_order_try
```
Plots:
```text
outputs/fa_tangent_hierarchy_first_order_try/first_order_prediction_scatter_by_early_step.png
outputs/fa_tangent_hierarchy_first_order_try/first_order_error_metrics_by_early_step.png
outputs/fa_tangent_hierarchy_first_order_try/first_order_gap_distribution_overlay.png
```
Summary:
| early s | predictor | MAE | bias | corr |
|---:|---|---:|---:|---:|
| 1 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
| 1 | linear velocity | 0.022925 | -0.020753 | 0.742632 |
| 2 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
| 2 | linear velocity | 0.012290 | -0.008418 | 0.815872 |
| 5 | fixed K0 | 0.023790 | +0.023790 | 0.941852 |
| 5 | linear velocity | 0.012232 | -0.012232 | 0.960370 |
| 5 | retangent | 0.016770 | +0.016770 | 0.971053 |
Interpretation:
```text
early average velocity improves magnitude substantially
```
At `s=2` or `s=5`, MAE is nearly cut in half relative to fixed `K0`.
But:
```text
linear extrapolation shifts from positive bias to negative bias
```
So the first-order average velocity captures the direction of operator drift,
but extrapolating it linearly to `T=50` overcorrects.
This is useful: the tangent-hierarchy route is real, but the model should not
be a naive global linear extrapolation.
## Experiment 2: Infinitesimal Derivative Probe
Script:
```text
scripts/fa_tangent_hierarchy_derivative_probe.py
```
Run:
```text
python scripts/fa_tangent_hierarchy_derivative_probe.py \
--width 64 \
--hidden-layers 2 \
--train-samples 128 \
--target-steps 50 \
--epsilon 0.05 \
--init-seeds 3 \
--feedback-seeds 6 \
--lr 1e-3 \
--torch-threads 8 \
--outdir outputs/fa_tangent_hierarchy_derivative_probe_eps005
```
Result:
```text
fixed MAE = 0.0237897
fixed bias = +0.0237897
derivative MAE = 270.362
derivative bias= -270.212
```
The infinitesimal first derivative is unusable when extrapolated linearly to
`T=50`.
Likely reasons:
```text
1. K_t is not globally linear for 50 steps;
2. the first derivative creates an indefinite operator path that can become
unstable when extrapolated;
3. ReLU gates and FA alignment dynamics make the actual early average velocity
very different from the infinitesimal derivative;
4. higher-order hierarchy terms are not small over this horizon.
```
This is not a normalization bug. The same setup shows fixed `K0` is good at
very short horizons and that early average velocity improves finite-time
magnitude.
## Consequence
The simple theorem:
```text
K_t = K_0 + t dot_K_0
```
is not enough for practical `T=50` prediction.
The better finite-time object is:
```text
K_t = K_0 + A_t
```
where `A_t` should be a bounded short-time alignment-gain path, not an
unbounded linear extrapolation.
Empirically, the useful object is:
```text
average early velocity over s=2 or s=5
```
not the infinitesimal `dot_K_0`.
## Revised Theory Direction
We should pursue one of these:
1. **Integrated short-time hierarchy**
Predict:
```text
K_s - K_0 = integral_0^s dot_K_u du
```
rather than only `dot_K_0`.
2. **Stable alignment-gain scalar**
Decompose:
```text
speed_FA,t = output_speed_t + hidden_alignment_gain_t
```
and model the hidden alignment gain as a bounded increasing process.
3. **Piecewise retangent estimator**
For empirical validation, use:
```text
measured K_s
```
as a conditional estimator, and do not claim it is architecture-only.
The most paper-safe claim:
```text
The exact e_0 theorem gives the initial operator cost. For finite time, the
successful predictor is an early-time integrated tangent-operator estimator,
which is an empirical finite-difference version of a truncated FA tangent
hierarchy. A pure infinitesimal first-derivative extrapolation is not stable
over nonlocal horizons.
```
This result is useful because it prevents us from overclaiming a too-simple
first-order theory.
|
