init commit

1 files changed, 243 insertions, 0 deletions

diff --git a/runs/slurm_logs/14363508_cifar100_depth.out b/runs/slurm_logs/14363508_cifar100_depth.out
new file mode 100644
index 0000000..2d44745
--- /dev/null
+++ b/runs/slurm_logs/14363508_cifar100_depth.out
@@ -0,0 +1,243 @@
+============================================================
+CIFAR-100 Depth Scaling Benchmark
+Job ID: 14363508 | Node: gpub039
+Start: Mon Dec 29 09:14:19 CST 2025
+============================================================
+NVIDIA A40, 46068 MiB
+============================================================
+================================================================================
+DEPTH SCALING BENCHMARK
+================================================================================
+Dataset: cifar100
+Depths: [4, 8, 12, 16, 20]
+Timesteps: 4
+Epochs: 150
+λ_reg: 0.3, λ_target: -0.1
+Device: cuda
+================================================================================
+
+Loading cifar100...
+Classes: 100, Input: (3, 32, 32)
+Train: 50000, Test: 10000
+
+Depth configurations: [(4, '4×1'), (8, '4×2'), (12, '4×3'), (16, '4×4'), (20, '4×5')]
+
+============================================================
+Depth = 4 conv layers (4 stages × 1 blocks)
+============================================================
+    Vanilla: depth=4, params=1,756,836
+      Epoch  10: train=0.494 test=0.423  σ=9.41e-01/3.56e-08
+      Epoch  20: train=0.626 test=0.503  σ=5.87e-01/2.43e-08
+      Epoch  30: train=0.703 test=0.550  σ=4.74e-01/1.97e-08
+      Epoch  40: train=0.755 test=0.564  σ=4.13e-01/1.70e-08
+      Epoch  50: train=0.797 test=0.542  σ=3.68e-01/1.50e-08
+      Epoch  60: train=0.830 test=0.581  σ=3.41e-01/1.42e-08
+      Epoch  70: train=0.862 test=0.583  σ=3.14e-01/1.29e-08
+      Epoch  80: train=0.883 test=0.599  σ=3.02e-01/1.21e-08
+      Epoch  90: train=0.905 test=0.594  σ=2.83e-01/1.09e-08
+      Epoch 100: train=0.920 test=0.607  σ=2.60e-01/9.94e-09
+      Epoch 110: train=0.932 test=0.611  σ=2.57e-01/9.62e-09
+      Epoch 120: train=0.941 test=0.610  σ=2.45e-01/9.26e-09
+      Epoch 130: train=0.949 test=0.616  σ=2.44e-01/8.78e-09
+      Epoch 140: train=0.951 test=0.613  σ=2.30e-01/8.58e-09
+      Epoch 150: train=0.952 test=0.614  σ=2.32e-01/8.66e-09
+      Best test acc: 0.621
+    Lyapunov: depth=4, params=1,756,836
+      Epoch  10: train=0.012 test=0.010 λ=1.940 σ=9.87e-02/2.73e-13
+      Epoch  20: train=0.010 test=0.010 λ=1.930 σ=3.70e-02/0.00e+00
+      Epoch  30: train=0.009 test=0.010 λ=1.920 σ=4.74e-03/0.00e+00
+      Epoch  40: train=0.009 test=0.010 λ=1.920 σ=2.81e-03/0.00e+00
+      Epoch  50: train=0.008 test=0.010 λ=1.920 
+      Epoch  60: train=0.008 test=0.010 λ=1.921 
+      Epoch  70: train=0.010 test=0.010 λ=1.922 
+      Epoch  80: train=0.009 test=0.010 λ=1.923 
+      Epoch  90: train=0.009 test=0.010 λ=1.919 
+      Epoch 100: train=0.009 test=0.010 λ=1.923 
+      Epoch 110: train=0.009 test=0.010 λ=1.921 
+      Epoch 120: train=0.009 test=0.010 λ=1.923 
+      Epoch 130: train=0.009 test=0.010 λ=1.923 
+      Epoch 140: train=0.009 test=0.010 λ=1.922 
+      Epoch 150: train=0.010 test=0.010 λ=1.921 
+      Best test acc: 0.054
+
+============================================================
+Depth = 8 conv layers (4 stages × 2 blocks)
+============================================================
+    Vanilla: depth=8, params=4,892,196
+      Epoch  10: train=0.390 test=0.338  σ=8.34e-01/3.11e-08
+      Epoch  20: train=0.547 test=0.438  σ=4.73e-01/2.15e-08
+      Epoch  30: train=0.633 test=0.454  σ=3.91e-01/1.80e-08
+      Epoch  40: train=0.699 test=0.489  σ=3.30e-01/1.55e-08
+      Epoch  50: train=0.754 test=0.509  σ=3.13e-01/1.41e-08
+      Epoch  60: train=0.795 test=0.503  σ=2.84e-01/1.27e-08
+      Epoch  70: train=0.836 test=0.511  σ=2.72e-01/1.18e-08
+      Epoch  80: train=0.869 test=0.517  σ=2.46e-01/1.02e-08
+      Epoch  90: train=0.897 test=0.523  σ=2.46e-01/1.00e-08
+      Epoch 100: train=0.917 test=0.519  σ=2.33e-01/9.01e-09
+      Epoch 110: train=0.932 test=0.528  σ=2.26e-01/8.65e-09
+      Epoch 120: train=0.947 test=0.537  σ=2.11e-01/8.16e-09
+      Epoch 130: train=0.953 test=0.526  σ=2.06e-01/7.74e-09
+      Epoch 140: train=0.954 test=0.538  σ=1.98e-01/7.35e-09
+      Epoch 150: train=0.956 test=0.522  σ=2.00e-01/7.34e-09
+      Best test acc: 0.541
+    Lyapunov: depth=8, params=4,892,196
+      Epoch  10: train=0.010 test=0.010 λ=2.704 σ=1.07e-01/7.42e-16
+      Epoch  20: train=0.009 test=0.010 λ=2.262 
+      Epoch  30: train=0.009 test=0.010 λ=2.272 
+      Epoch  40: train=0.009 test=0.010 λ=2.264 σ=6.25e-03/0.00e+00
+      Epoch  50: train=0.008 test=0.010 λ=2.281 
+      Epoch  60: train=0.009 test=0.010 λ=2.273 
+      Epoch  70: train=0.008 test=0.010 λ=2.267 
+      Epoch  80: train=0.009 test=0.010 λ=2.263 
+      Epoch  90: train=0.009 test=0.010 λ=2.264 
+      Epoch 100: train=0.009 test=0.010 λ=2.261 
+      Epoch 110: train=0.008 test=0.010 λ=2.264 
+      Epoch 120: train=0.009 test=0.010 λ=2.261 
+      Epoch 130: train=0.009 test=0.010 λ=2.264 
+      Epoch 140: train=0.009 test=0.010 λ=2.263 
+      Epoch 150: train=0.010 test=0.010 λ=2.262 
+      Best test acc: 0.028
+
+============================================================
+Depth = 12 conv layers (4 stages × 3 blocks)
+============================================================
+    Vanilla: depth=12, params=8,027,556
+      Epoch  10: train=0.215 test=0.064  σ=6.68e-01/2.31e-08
+      Epoch  20: train=0.286 test=0.052  σ=3.31e-01/1.58e-08
+      Epoch  30: train=0.336 test=0.081  σ=2.75e-01/1.39e-08
+      Epoch  40: train=0.369 test=0.069  σ=2.34e-01/1.27e-08
+      Epoch  50: train=0.410 test=0.064  σ=2.31e-01/1.24e-08
+      Epoch  60: train=0.435 test=0.059  σ=2.20e-01/1.22e-08
+      Epoch  70: train=0.262 test=0.108  σ=2.07e-01/1.12e-08
+      Epoch  80: train=0.390 test=0.110  σ=2.14e-01/1.20e-08
+      Epoch  90: train=0.437 test=0.106  σ=2.23e-01/1.22e-08
+      Epoch 100: train=0.473 test=0.124  σ=2.28e-01/1.25e-08
+      Epoch 110: train=0.500 test=0.103  σ=2.28e-01/1.24e-08
+      Epoch 120: train=0.527 test=0.095  σ=2.35e-01/1.25e-08
+      Epoch 130: train=0.536 test=0.107  σ=2.38e-01/1.28e-08
+      Epoch 140: train=0.545 test=0.111  σ=2.40e-01/1.26e-08
+      Epoch 150: train=0.547 test=0.102  σ=2.40e-01/1.29e-08
+      Best test acc: 0.126
+    Lyapunov: depth=12, params=8,027,556
+      Epoch  10: train=0.013 test=0.010 λ=2.873 σ=2.57e-01/2.15e-13
+      Epoch  20: train=0.010 test=0.010 λ=2.629 σ=2.81e-02/0.00e+00
+      Epoch  30: train=0.009 test=0.010 λ=2.465 σ=6.68e-03/0.00e+00
+      Epoch  40: train=0.009 test=0.010 λ=2.480 
+      Epoch  50: train=0.009 test=0.010 λ=2.470 
+      Epoch  60: train=0.009 test=0.010 λ=2.482 
+      Epoch  70: train=0.008 test=0.010 λ=2.473 
+      Epoch  80: train=0.008 test=0.010 λ=2.463 
+      Epoch  90: train=0.008 test=0.010 λ=2.465 
+      Epoch 100: train=0.009 test=0.010 λ=2.463 
+      Epoch 110: train=0.008 test=0.010 λ=2.470 
+      Epoch 120: train=0.009 test=0.010 λ=2.468 
+      Epoch 130: train=0.010 test=0.010 λ=2.470 
+      Epoch 140: train=0.009 test=0.010 λ=2.463 
+      Epoch 150: train=0.010 test=0.010 λ=2.462 
+      Best test acc: 0.011
+
+============================================================
+Depth = 16 conv layers (4 stages × 4 blocks)
+============================================================
+    Vanilla: depth=16, params=11,162,916
+      Epoch  10: train=0.094 test=0.011  σ=4.41e-01/1.38e-08
+      Epoch  20: train=0.134 test=0.020  σ=2.83e-01/1.10e-08
+      Epoch  30: train=0.156 test=0.022  σ=2.27e-01/9.71e-09
+      Epoch  40: train=0.174 test=0.022  σ=1.97e-01/9.00e-09
+      Epoch  50: train=0.184 test=0.022  σ=1.79e-01/8.89e-09
+      Epoch  60: train=0.198 test=0.021  σ=1.70e-01/8.88e-09
+      Epoch  70: train=0.212 test=0.022  σ=1.60e-01/8.82e-09
+      Epoch  80: train=0.224 test=0.027  σ=1.63e-01/8.93e-09
+      Epoch  90: train=0.235 test=0.031  σ=1.57e-01/8.95e-09
+      Epoch 100: train=0.241 test=0.032  σ=1.60e-01/9.14e-09
+      Epoch 110: train=0.255 test=0.037  σ=1.58e-01/9.23e-09
+      Epoch 120: train=0.259 test=0.034  σ=1.61e-01/9.22e-09
+      Epoch 130: train=0.263 test=0.038  σ=1.61e-01/9.35e-09
+      Epoch 140: train=0.265 test=0.032  σ=1.63e-01/9.35e-09
+      Epoch 150: train=0.269 test=0.037  σ=1.65e-01/9.36e-09
+      Best test acc: 0.040
+    Lyapunov: depth=16, params=11,162,916
+      Epoch  10: train=0.013 test=0.010 λ=2.901 σ=2.73e-01/2.05e-13
+      Epoch  20: train=0.009 test=0.010 λ=3.238 σ=1.03e-02/0.00e+00
+      Epoch  30: train=0.009 test=0.010 λ=2.605 σ=3.07e-03/0.00e+00
+      Epoch  40: train=0.008 test=0.010 λ=2.603 
+      Epoch  50: train=0.008 test=0.010 λ=2.610 
+      Epoch  60: train=0.009 test=0.010 λ=2.627 
+      Epoch  70: train=0.009 test=0.010 λ=2.609 
+      Epoch  80: train=0.009 test=0.010 λ=2.607 
+      Epoch  90: train=0.009 test=0.010 λ=2.622 
+      Epoch 100: train=0.009 test=0.010 λ=2.614 
+      Epoch 110: train=0.009 test=0.010 λ=2.606 
+      Epoch 120: train=0.009 test=0.010 λ=2.602 
+      Epoch 130: train=0.009 test=0.010 λ=2.615 
+      Epoch 140: train=0.010 test=0.010 λ=2.602 
+      Epoch 150: train=0.010 test=0.010 λ=2.603 
+      Best test acc: 0.011
+
+============================================================
+Depth = 20 conv layers (4 stages × 5 blocks)
+============================================================
+    Vanilla: depth=20, params=14,298,276
+      Epoch  10: train=0.010 test=0.011  σ=3.06e+00/4.22e-08
+      Epoch  20: train=0.010 test=0.010  σ=2.15e+00/2.95e-08
+      Epoch  30: train=0.010 test=0.010  σ=7.74e-01/2.37e-11
+      Epoch  40: train=0.009 test=0.010  σ=1.44e-01/0.00e+00
+      Epoch  50: train=0.009 test=0.010  σ=1.51e-02/0.00e+00
+      Epoch  60: train=0.025 test=0.010  σ=2.05e-01/1.31e-11
+      Epoch  70: train=0.032 test=0.010  σ=1.80e-01/1.69e-09
+      Epoch  80: train=0.040 test=0.010  σ=1.61e-01/1.82e-09
+      Epoch  90: train=0.043 test=0.010  σ=1.51e-01/2.04e-09
+      Epoch 100: train=0.046 test=0.011  σ=1.49e-01/2.28e-09
+      Epoch 110: train=0.050 test=0.011  σ=1.56e-01/2.59e-09
+      Epoch 120: train=0.049 test=0.012  σ=1.53e-01/2.89e-09
+      Epoch 130: train=0.053 test=0.010  σ=1.51e-01/3.14e-09
+      Epoch 140: train=0.055 test=0.010  σ=1.49e-01/3.28e-09
+      Epoch 150: train=0.053 test=0.010  σ=1.51e-01/3.29e-09
+      Best test acc: 0.012
+    Lyapunov: depth=20, params=14,298,276
+      Epoch  10: train=0.013 test=0.010 λ=2.968 σ=3.33e-01/5.32e-13
+      Epoch  20: train=0.011 test=0.010 λ=2.969 σ=5.00e-02/2.54e-43
+      Epoch  30: train=0.008 test=0.010 λ=2.719 σ=1.06e-02/0.00e+00
+      Epoch  40: train=0.009 test=0.010 λ=2.737 
+      Epoch  50: train=0.009 test=0.010 λ=2.729 
+      Epoch  60: train=0.009 test=0.010 λ=2.748 
+      Epoch  70: train=0.009 test=0.010 λ=2.740 
+      Epoch  80: train=0.008 test=0.010 λ=2.721 
+      Epoch  90: train=0.009 test=0.010 λ=2.763 
+      Epoch 100: train=0.008 test=0.010 λ=2.735 
+      Epoch 110: train=0.009 test=0.010 λ=2.716 
+      Epoch 120: train=0.009 test=0.010 λ=2.718 
+      Epoch 130: train=0.009 test=0.010 λ=2.756 
+      Epoch 140: train=0.009 test=0.010 λ=2.726 
+      Epoch 150: train=0.010 test=0.010 λ=2.714 
+      Best test acc: 0.016
+
+====================================================================================================
+DEPTH SCALING RESULTS: CIFAR100
+====================================================================================================
+Depth    Vanilla Acc  Lyapunov Acc Δ Acc    Lyap λ     Van ∇norm    Lyap ∇norm   Van κ     
+----------------------------------------------------------------------------------------------------
+4        0.614        0.010        -0.604   1.921      4.57e-01     8.82e-02     1.2e+09   
+8        0.522        0.010        -0.512   2.262      3.86e-01     8.73e-02     1.4e+09   
+12       0.102        0.010        -0.092   2.462      6.74e-01     8.77e-02     2.5e+07   
+16       0.037        0.010        -0.027   2.603      5.04e-01     8.77e-02     2.4e+07   
+20       0.010        0.010        -0.000   2.714      2.96e-01     8.80e-02     6.5e+07   
+====================================================================================================
+
+GRADIENT HEALTH ANALYSIS:
+  Depth 4: ⚠️ Vanilla has ill-conditioned gradients (κ > 1e6)
+  Depth 8: ⚠️ Vanilla has ill-conditioned gradients (κ > 1e6)
+  Depth 12: ⚠️ Vanilla has ill-conditioned gradients (κ > 1e6)
+  Depth 16: ⚠️ Vanilla has ill-conditioned gradients (κ > 1e6)
+  Depth 20: ⚠️ Vanilla has ill-conditioned gradients (κ > 1e6)
+
+
+KEY OBSERVATIONS:
+  Vanilla  4→20 layers: -0.604 accuracy change
+  Lyapunov 4→20 layers: +0.000 accuracy change
+  ✓ Lyapunov regularization enables better depth scaling!
+
+Results saved to runs/depth_scaling/cifar100_20251230-213033
+============================================================
+Finished: Tue Dec 30 21:30:34 CST 2025
+============================================================