summaryrefslogtreecommitdiff
path: root/notes/01_theory_notes.md
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-05-29 08:43:19 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-05-29 08:43:19 -0500
commitdd14582dcfd3b6e3e7b0e28f66bb0f2994f106c4 (patch)
tree99c06ef0e6bb26a040409408040e1e88bc14f69f /notes/01_theory_notes.md
parentc47a74792e6ee78181b087ead595465140b77825 (diff)
Add multilayer capacity distribution matching
Diffstat (limited to 'notes/01_theory_notes.md')
-rw-r--r--notes/01_theory_notes.md42
1 files changed, 42 insertions, 0 deletions
diff --git a/notes/01_theory_notes.md b/notes/01_theory_notes.md
index 7a54374..b22e535 100644
--- a/notes/01_theory_notes.md
+++ b/notes/01_theory_notes.md
@@ -166,6 +166,48 @@ C_l\approx c/2,
C_{\mathrm{all}}=\Theta(L).
\]
+## Candidate Corollary 2: Observed Capacity Distribution
+
+The threshold cost \(C_l(q)\) is deterministic once \(q\) is chosen. For empirical distribution matching, define the observed alignment surprisal:
+
+\[
+S_l
+=
+-\log \Pr(Q_l'\ge Q_l),
+\]
+
+where \(Q_l'\) is an independent draw from the same beta law as \(Q_l\). If:
+
+\[
+Q_l\sim \mathrm{Beta}\left(\frac12,\frac{D_l-1}{2}\right),
+\]
+
+and the survival function is continuous, then by the probability integral transform:
+
+\[
+U_l=\Pr(Q_l'\ge Q_l)\sim \mathrm{Uniform}(0,1),
+\]
+
+so:
+
+\[
+S_l=-\log U_l\sim \mathrm{Exp}(1).
+\]
+
+For \(L\) independent feedback-aligned layers, the observed total capacity surprisal is:
+
+\[
+S_{1:L}=\sum_{l=1}^L S_l.
+\]
+
+If the layerwise feedback matrices are independent, then:
+
+\[
+S_{1:L}\sim \mathrm{Gamma}(L,1).
+\]
+
+This is useful because it predicts a full distribution rather than only a mean or bound. It also shows that after the beta-tail transform, the null distribution of observed capacity surprisal is dimension-free; dimension enters through the raw \(Q_l\) scale and the threshold cost \(C_l(q)\), not through the transformed surprisal law.
+
## Candidate Theorem 2: Prior-Free Minimax Bound
Let: