diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-05-29 08:43:19 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-05-29 08:43:19 -0500 |
| commit | dd14582dcfd3b6e3e7b0e28f66bb0f2994f106c4 (patch) | |
| tree | 99c06ef0e6bb26a040409408040e1e88bc14f69f /notes/01_theory_notes.md | |
| parent | c47a74792e6ee78181b087ead595465140b77825 (diff) | |
Add multilayer capacity distribution matching
Diffstat (limited to 'notes/01_theory_notes.md')
| -rw-r--r-- | notes/01_theory_notes.md | 42 |
1 files changed, 42 insertions, 0 deletions
diff --git a/notes/01_theory_notes.md b/notes/01_theory_notes.md index 7a54374..b22e535 100644 --- a/notes/01_theory_notes.md +++ b/notes/01_theory_notes.md @@ -166,6 +166,48 @@ C_l\approx c/2, C_{\mathrm{all}}=\Theta(L). \] +## Candidate Corollary 2: Observed Capacity Distribution + +The threshold cost \(C_l(q)\) is deterministic once \(q\) is chosen. For empirical distribution matching, define the observed alignment surprisal: + +\[ +S_l += +-\log \Pr(Q_l'\ge Q_l), +\] + +where \(Q_l'\) is an independent draw from the same beta law as \(Q_l\). If: + +\[ +Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right), +\] + +and the survival function is continuous, then by the probability integral transform: + +\[ +U_l=\Pr(Q_l'\ge Q_l)\sim \mathrm{Uniform}(0,1), +\] + +so: + +\[ +S_l=-\log U_l\sim \mathrm{Exp}(1). +\] + +For \(L\) independent feedback-aligned layers, the observed total capacity surprisal is: + +\[ +S_{1:L}=\sum_{l=1}^L S_l. +\] + +If the layerwise feedback matrices are independent, then: + +\[ +S_{1:L}\sim \mathrm{Gamma}(L,1). +\] + +This is useful because it predicts a full distribution rather than only a mean or bound. It also shows that after the beta-tail transform, the null distribution of observed capacity surprisal is dimension-free; dimension enters through the raw \(Q_l\) scale and the threshold cost \(C_l(q)\), not through the transformed surprisal law. + ## Candidate Theorem 2: Prior-Free Minimax Bound Let: |
