diff options
27 files changed, 100 insertions, 9 deletions
@@ -1,16 +1,40 @@ # SDIL — results log (seed 0 unless noted; runs on timan107 GTX-1080 / ep_pascal) +## 2026-07-21 audit and version boundary + +Git was initialized after inheriting the project. The original code/results are preserved at +`215acff`. Four audit findings changed the canonical implementation: + +1. `49b86d6`: residual-block local updates now include the branch Jacobian + `res_alpha=1/sqrt(depth)`. The omission preserved cosine direction but inflated update norms by + `sqrt(depth)`, confounding cross-depth optimization comparisons. +2. `5255166`: the Harnett baseline is now a per-neuron affine soma-dendrite fit + `ahat_l = p_l * h_l + b_l`, rather than a dense population map. This follows the paper's + per-cell least-squares somatic-event to dendritic-event relation. +3. `c2769e6`: causal perturbation targets and predicted residuals are measured on the same + pre-update network state. +4. `0f107d1` / `e0bfdfd`: simultaneous hidden-layer perturbations reduce calibration from + quadratic to linear depth work; batching directions makes the reduction visible in GPU wall + time. `66e3034` fixes diagnostics so raw-apical ablations report the signal actually used. + +All older `deepres_*` files predate item 1. Their alignment directions remain informative, but +their accuracy and stability should not be pooled with the corrected runs below. + ## Method SDIL (Somato-Dendritic Innovation Learning), a local non-backprop rule inspired by Harnett 2026. Per hidden layer l: - forward: `u_l = W_l h_{l-1}`, `h_l = φ(u_l)` - apical feedback `a_l = A_l c` (c = output error e = softmax−onehot, broadcast) -- predictor `ĥ_l = P_l h_l`; teaching signal (innovation) `r_l = a_l − ĥ_l` +- default per-neuron predictor `ĥ_l = p_l ⊙ h_l + b_l`; teaching signal `r_l = a_l − ĥ_l` - three-factor update `ΔW_l = η (r_l ⊙ φ'(u_l)) h_{l-1}^T` - `A_l` LEARNED by amortized node perturbation (q_l ≈ −∇_{h_l}L, forward-only, no weight transport) - `P_l` learned on neutral (c=0) periods only, so it strips the soma-predictable nuisance without eating the teaching signal. +The scalable perturbation mode injects independent Rademacher interventions at all hidden layers +in the same antithetic forward evaluations. Cross-layer interference adds variance but averages to +zero; it trains all `A_l` using `O(depth)` rather than `O(depth^2)` forward work. + Distinct from the failed `~/sdrn` (fixed-random feedback + post-hoc residualization) and from Dual Prop / DFA / FA: the feedback pathway is *learned by causal perturbation*, so the residual is aligned by construction rather than being a random projection. @@ -22,7 +46,45 @@ magnitude-invariant so alignment looked fine while optimization blew up. Fixing "worse than DFA everywhere" to "matches DFA on MNIST, beats it on FMNIST/CIFAR". (Codex independently flagged the same σ-scale issue.) -## Accuracy (test %, seed 0) +## Corrected narrow depth pilot (CIFAR-10, width 64, 5 epochs, seed 0) + +| depth | method | test acc | early-third alignment | wall time | +|------:|--------|---------:|----------------------:|----------:| +| 30 | BP | 41.31% | — | 29.4 s | +| 30 | DFA | 37.31% | 0.084 | 19.4 s | +| 30 | SDIL layerwise | 42.01% | 0.371 | 339.5 s | +| 30 | SDIL multiplexed+batched | 41.46% | 0.370 | 47.3 s | +| 60 | BP | 42.93% | — | 63.7 s | +| 60 | DFA | 37.31% | 0.041 | 41.1 s | +| 60 | SDIL layerwise | 40.79% | 0.328 | 1238.5 s | +| 60 | SDIL multiplexed+batched | 41.10% | 0.332 | 90.3 s | + +Narrowing the model exposed the performance difference that width-256 flattened CIFAR had hidden. +DFA alignment halves from depth 30 to 60, while both SDIL estimators remain near 0.33–0.37. Batched +multiplexing preserves alignment while accelerating SDIL 7.2x at d30 and 13.7x at d60. Its wall +growth from d30 to d60 is 1.91x, versus 3.65x for layerwise calibration. These are pilot, single-seed +results; the ~0.5-point layerwise/multiplexed accuracy difference is not yet statistically resolved. + +## Harnett-specific residualization (MNIST, 15 epochs, seed 0) + +| nuisance rho | raw apical acc | residual acc | raw teaching alignment | residual alignment | +|-------------:|---------------:|-------------:|-----------------------:|-------------------:| +| 0.00 | 97.31% | 97.31% | 0.138 | 0.138 | +| 0.02 | 96.90% | 97.27% | 0.096 | 0.156 | +| 0.05 | 96.48% | 97.24% | 0.092 | 0.160 | +| 0.10 | 95.87% | 97.32% | 0.068 | 0.140 | +| 0.20 | 95.60% | 97.31% | 0.103 | 0.147 | +| 0.50 | 8.92% | 97.18% | 0.000 | 0.149 | +| 1.00 | 8.92% | 97.29% | 0.000 | 0.140 | +| 2.00 | 9.82% | 97.31% | 0.000 | 0.121 | + +At zero nuisance, subtraction is a no-op. As normal soma-dendrite coupling grows, raw feedback +degrades continuously and eventually enters a self-reinforcing Hebbian failure, while the learned +innovation remains stable. The low-rho sweep is important: it shows a graded 0.37→1.71 point gap +before the strong-contamination phase transition, rather than only an artificial chance-level +contrast. + +## Legacy accuracy (pre-audit implementation) | task | BP | DFA | SDIL | |------------------------------|-----------|-------|---------------| | MNIST d3 / d5 / d7 / d10 | 98.3 | 97.3 / 97.2 / 97.1 / 96.8 | 97.4 / 97.3 / 97.1 / 97.1 | @@ -33,14 +95,14 @@ SDIL matches DFA on easy MNIST and **beats DFA on FashionMNIST (+1–2 pts) and On CIFAR the SDIL−DFA gap GROWS with depth (+1.8 → +2.7 → +5.0 at d5/10/20): DFA degrades with depth, SDIL holds. -## Credit assignment / depth utility — the mechanistic result +## Legacy credit assignment / depth utility Per-layer alignment cos(r_l, −∇_{h_l}L), early-third average, CIFAR no-BN residual: | depth | DFA early-align | SDIL early-align | |-------|-----------------|------------------| | 5 | 0.55 | 0.83 | | 10 | 0.36 | 0.86 | | 20 | 0.25 | 0.88 | -| 30 | 0.19 | (running) | +| 30 | 0.19 | 0.87 | DFA's teaching signal to early/deep layers collapses toward random (~0.05–0.19) as depth grows; SDIL stays ~0.85 and is depth-invariant. This is the "does credit reach early layers" metric that @@ -61,8 +123,11 @@ Analysis: `experiments/analyze_depth.py {depth,deepres}` → tables + figures in Smoke/mechanism checks: `experiments/smoke.py`. ## Open items -- Complete CIFAR SDIL d30 (running) + seeds 1,2 (running) for mean±std. -- Codex-suggested boosts (not yet applied): separate normalized feedback direction from a learned - per-layer scalar gain; Adam-like second-moment on local updates; forward-only gradient-norm gain - calibration. Current results already beat DFA without these. -- GRAPE (ICLR2026) read for DFA-failure regimes only; treated as unverified, not a baseline. +- Repeat corrected width-64 d30/d60 and nuisance transition runs for seeds 1–4. +- Compare multiplexed calibration at fixed loss-evaluation budgets and sweep directions (4/8/16/32). +- Add a magnitude-matched or clipped raw-apical control to separate directional contamination from + simple update explosion at large nuisance. +- Validate simultaneous perturbation on a genuinely depth-necessary compositional task and a CNN, + not only flattened CIFAR. +- FA's residual-network implementation and the current EP prototype are not yet strong enough for a + publication-quality scaling comparison; do not overstate those baselines. diff --git a/results/diag_nuis_low_rho0p02_res0_s0.json b/results/diag_nuis_low_rho0p02_res0_s0.json new file mode 100644 index 0000000..24c2dd1 --- /dev/null +++ b/results/diag_nuis_low_rho0p02_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.02, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p02_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.01925944909453392, -0.02566162496805191, 0.03369072079658508], "cos_apical_negg": [-0.019273214042186737, -0.025740373879671097, 0.03344225510954857], "cos_Ac_negg": [-0.019259434193372726, -0.02566162496805191, 0.03369072079658508], "r_norm": [4.897989273071289, 4.826717376708984, 4.823838233947754], "ldr": {"dL_sdil": -0.009076356887817383, "dL_bp": -0.2385098934173584, "ratio": 0.03805442515516557}}, {"step": 100, "epoch": 0, "train_loss": 0.206071138381958, "cos_r_negg": [0.4681476652622223, 0.4429267644882202, 0.38471874594688416], "cos_apical_negg": [0.26263806223869324, 0.2645108997821808, 0.2631727457046509], "cos_Ac_negg": [0.46821776032447815, 0.4429766535758972, 0.3847416937351227], "r_norm": [0.8438946008682251, 0.8258217573165894, 0.8231493830680847]}, {"step": 200, "epoch": 0, "train_loss": 0.33237993717193604, "cos_r_negg": [0.48675966262817383, 0.4554031491279602, 0.42761093378067017], "cos_apical_negg": [0.2616196870803833, 0.2735486626625061, 0.2730584144592285], "cos_Ac_negg": [0.4869941473007202, 0.4553951025009155, 0.4277513027191162], "r_norm": [0.7278070449829102, 0.707707405090332, 0.7060608863830566]}, {"step": 300, "epoch": 0, "train_loss": 0.19734743237495422, "cos_r_negg": [0.5285393595695496, 0.52535480260849, 0.4794597029685974], "cos_apical_negg": [0.23711776733398438, 0.29444169998168945, 0.2775915265083313], "cos_Ac_negg": [0.5286089181900024, 0.5253909826278687, 0.4794451594352722], "r_norm": [0.6054061651229858, 0.591116726398468, 0.5910351872444153]}, {"step": 400, "epoch": 0, "train_loss": 0.15375865995883942, "cos_r_negg": [0.5338476300239563, 0.5296660661697388, 0.4938103258609772], "cos_apical_negg": [0.2340439260005951, 0.28611883521080017, 0.33072173595428467], "cos_Ac_negg": [0.5339211821556091, 0.5297133326530457, 0.49383288621902466], "r_norm": [0.5766323804855347, 0.5654773712158203, 0.5646674036979675]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9449, "test_loss": 0.19892386169433593}, {"step": 500, "epoch": 1, "train_loss": 0.1472272425889969, "cos_r_negg": [0.526983380317688, 0.5274121761322021, 0.49484068155288696], "cos_apical_negg": [0.2006608545780182, 0.2437020242214203, 0.28972962498664856], "cos_Ac_negg": [0.5270229578018188, 0.5274916887283325, 0.49490851163864136], "r_norm": [0.4700573980808258, 0.4561854600906372, 0.4551491439342499], "ldr": {"dL_sdil": -0.007757142186164856, "dL_bp": -0.006305679678916931, "ratio": 1.2301833555074002}}, {"step": 600, "epoch": 1, "train_loss": 0.07822880148887634, "cos_r_negg": [0.5186655521392822, 0.5267373323440552, 0.45328375697135925], "cos_apical_negg": [0.20669317245483398, 0.2607686519622803, 0.3002426028251648], "cos_Ac_negg": [0.518663763999939, 0.5267905592918396, 0.4531635642051697], "r_norm": [0.48179900646209717, 0.4725462794303894, 0.47014787793159485]}, {"step": 700, "epoch": 1, "train_loss": 0.21658703684806824, "cos_r_negg": [0.5297616720199585, 0.5559932589530945, 0.48670634627342224], "cos_apical_negg": [0.19816885888576508, 0.2939818799495697, 0.3034880459308624], "cos_Ac_negg": [0.5298179388046265, 0.556054949760437, 0.48665502667427063], "r_norm": [0.46541786193847656, 0.4562772512435913, 0.45401731133461]}, {"step": 800, "epoch": 1, "train_loss": 0.1682123839855194, "cos_r_negg": [0.5429261922836304, 0.5628002285957336, 0.5304246544837952], "cos_apical_negg": [0.20251761376857758, 0.2674803137779236, 0.3079812228679657], "cos_Ac_negg": [0.542924165725708, 0.5628063678741455, 0.5304079055786133], "r_norm": [0.419233500957489, 0.40934354066848755, 0.40891626477241516]}, {"step": 900, "epoch": 1, "train_loss": 0.20421729981899261, "cos_r_negg": [0.5431697368621826, 0.579277753829956, 0.5153729915618896], "cos_apical_negg": [0.19254359602928162, 0.26994097232818604, 0.32042673230171204], "cos_Ac_negg": [0.5431655645370483, 0.5792593955993652, 0.5153427720069885], "r_norm": [0.4034919738769531, 0.392557829618454, 0.3928130269050598]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9518, "test_loss": 0.1722327033996582}, {"step": 1000, "epoch": 2, "train_loss": 0.041068803519010544, "cos_r_negg": [0.5078740119934082, 0.550865888595581, 0.5254982113838196], "cos_apical_negg": [0.14767752587795258, 0.22598004341125488, 0.2837088406085968], "cos_Ac_negg": [0.507811427116394, 0.5508331656455994, 0.5254217982292175], "r_norm": [0.33645105361938477, 0.3285433053970337, 0.3294784724712372], "ldr": {"dL_sdil": -0.006873175501823425, "dL_bp": -0.005714923143386841, "ratio": 1.202671554695689}}, {"step": 1100, "epoch": 2, "train_loss": 0.11744073033332825, "cos_r_negg": [0.5406942367553711, 0.5750820636749268, 0.5276658535003662], "cos_apical_negg": [0.17650026082992554, 0.260485976934433, 0.3302422761917114], "cos_Ac_negg": [0.5406332015991211, 0.5750493407249451, 0.5275596380233765], "r_norm": [0.35915249586105347, 0.35092929005622864, 0.3507956266403198]}, {"step": 1200, "epoch": 2, "train_loss": 0.1697431206703186, "cos_r_negg": [0.48784440755844116, 0.5374298095703125, 0.5122506618499756], "cos_apical_negg": [0.16910775005817413, 0.23454412817955017, 0.3069744110107422], "cos_Ac_negg": [0.48777857422828674, 0.5373592376708984, 0.5121716856956482], "r_norm": [0.41340339183807373, 0.40443333983421326, 0.4033299386501312]}, {"step": 1300, "epoch": 2, "train_loss": 0.14309418201446533, "cos_r_negg": [0.4753113389015198, 0.5426746010780334, 0.49981796741485596], "cos_apical_negg": [0.17476284503936768, 0.23367735743522644, 0.27802371978759766], "cos_Ac_negg": [0.47528213262557983, 0.5426498651504517, 0.49971646070480347], "r_norm": [0.3467801511287689, 0.3394143879413605, 0.337238073348999]}, {"step": 1400, "epoch": 2, "train_loss": 0.1667533814907074, "cos_r_negg": [0.45972293615341187, 0.5054548382759094, 0.5182697772979736], "cos_apical_negg": [0.13602866232395172, 0.207316055893898, 0.26495563983917236], "cos_Ac_negg": [0.45968353748321533, 0.505387544631958, 0.5181490182876587], "r_norm": [0.31786465644836426, 0.3079378008842468, 0.30868756771087646]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9631, "test_loss": 0.134154386138916}, {"step": 1500, "epoch": 3, "train_loss": 0.06395118683576584, "cos_r_negg": [0.4347681999206543, 0.4764384925365448, 0.4741770029067993], "cos_apical_negg": [0.13339635729789734, 0.20873010158538818, 0.2309040129184723], "cos_Ac_negg": [0.4347454607486725, 0.47638610005378723, 0.47409844398498535], "r_norm": [0.3315296173095703, 0.32385939359664917, 0.3227711319923401], "ldr": {"dL_sdil": -0.005825206637382507, "dL_bp": -0.005550175905227661, "ratio": 1.0495535162941048}}, {"step": 1600, "epoch": 3, "train_loss": 0.09140448272228241, "cos_r_negg": [0.4630548357963562, 0.5250738859176636, 0.5203036069869995], "cos_apical_negg": [0.13214173913002014, 0.22044754028320312, 0.2836686074733734], "cos_Ac_negg": [0.4630080461502075, 0.5249927043914795, 0.5201841592788696], "r_norm": [0.3129541277885437, 0.3022066354751587, 0.30283039808273315]}, {"step": 1700, "epoch": 3, "train_loss": 0.06026475504040718, "cos_r_negg": [0.43740707635879517, 0.49217793345451355, 0.4837079644203186], "cos_apical_negg": [0.1464240401983261, 0.21427947282791138, 0.29652056097984314], "cos_Ac_negg": [0.43735408782958984, 0.4920836091041565, 0.4835726320743561], "r_norm": [0.3232586681842804, 0.3133507966995239, 0.31499576568603516]}, {"step": 1800, "epoch": 3, "train_loss": 0.038792576640844345, "cos_r_negg": [0.4113686680793762, 0.46835997700691223, 0.48693421483039856], "cos_apical_negg": [0.11118613183498383, 0.18830563127994537, 0.23896212875843048], "cos_Ac_negg": [0.4112994074821472, 0.46827012300491333, 0.4868676960468292], "r_norm": [0.24922126531600952, 0.2421068549156189, 0.2417508065700531]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9657, "test_loss": 0.12998302383422852}, {"step": 1900, "epoch": 4, "train_loss": 0.09116171300411224, "cos_r_negg": [0.4054185748100281, 0.458378404378891, 0.4776462912559509], "cos_apical_negg": [0.12753790616989136, 0.20597206056118011, 0.21349479258060455], "cos_Ac_negg": [0.40535789728164673, 0.45829713344573975, 0.47761425375938416], "r_norm": [0.27217990159988403, 0.2658555209636688, 0.2657337486743927]}, {"step": 2000, "epoch": 4, "train_loss": 0.16241157054901123, "cos_r_negg": [0.40267765522003174, 0.4554517865180969, 0.4866638779640198], "cos_apical_negg": [0.09805499017238617, 0.18514610826969147, 0.2427842617034912], "cos_Ac_negg": [0.40264204144477844, 0.45538997650146484, 0.4865497946739197], "r_norm": [0.2589050829410553, 0.2514130771160126, 0.2510063052177429], "ldr": {"dL_sdil": -0.005476370453834534, "dL_bp": -0.00536104291677475, "ratio": 1.0215121458361998}}, {"step": 2100, "epoch": 4, "train_loss": 0.01772943325340748, "cos_r_negg": [0.44070112705230713, 0.4932866096496582, 0.553056001663208], "cos_apical_negg": [0.16076679527759552, 0.2079952359199524, 0.3001341223716736], "cos_Ac_negg": [0.4406587481498718, 0.4932495951652527, 0.5529830455780029], "r_norm": [0.2952989637851715, 0.2919972836971283, 0.28878647089004517]}, {"step": 2200, "epoch": 4, "train_loss": 0.057878486812114716, "cos_r_negg": [0.3757783770561218, 0.42753899097442627, 0.45231103897094727], "cos_apical_negg": [0.10571407526731491, 0.1683899313211441, 0.2443653643131256], "cos_Ac_negg": [0.3757992684841156, 0.42747724056243896, 0.4522625207901001], "r_norm": [0.287088543176651, 0.27858322858810425, 0.2778674364089966]}, {"step": 2300, "epoch": 4, "train_loss": 0.22512993216514587, "cos_r_negg": [0.36532062292099, 0.42891421914100647, 0.4331624507904053], "cos_apical_negg": [0.09902752935886383, 0.14983227849006653, 0.23736990988254547], "cos_Ac_negg": [0.3652558922767639, 0.4288051426410675, 0.43303361535072327], "r_norm": [0.2867211103439331, 0.27858883142471313, 0.27877819538116455]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9668, "test_loss": 0.1339680564880371}, {"step": 2400, "epoch": 5, "train_loss": 0.08477146923542023, "cos_r_negg": [0.41073328256607056, 0.4612548351287842, 0.4929535984992981], "cos_apical_negg": [0.08546945452690125, 0.16547246277332306, 0.2646588683128357], "cos_Ac_negg": [0.410677433013916, 0.4611518383026123, 0.49284693598747253], "r_norm": [0.2526025176048279, 0.24683891236782074, 0.24579359591007233]}, {"step": 2500, "epoch": 5, "train_loss": 0.025092322379350662, "cos_r_negg": [0.3764820694923401, 0.43139785528182983, 0.4772835373878479], "cos_apical_negg": [0.08639393001794815, 0.1544605791568756, 0.24907028675079346], "cos_Ac_negg": [0.37641221284866333, 0.43128007650375366, 0.4771392345428467], "r_norm": [0.21813178062438965, 0.21333324909210205, 0.21203911304473877], "ldr": {"dL_sdil": -0.0030814483761787415, "dL_bp": -0.002756349742412567, "ratio": 1.1179453495192606}}, {"step": 2600, "epoch": 5, "train_loss": 0.06760348379611969, "cos_r_negg": [0.38867685198783875, 0.448198139667511, 0.47526270151138306], "cos_apical_negg": [0.09218351542949677, 0.14963187277317047, 0.2465364933013916], "cos_Ac_negg": [0.38863271474838257, 0.44811707735061646, 0.4751494526863098], "r_norm": [0.25988900661468506, 0.25497108697891235, 0.2504923939704895]}, {"step": 2700, "epoch": 5, "train_loss": 0.06910201907157898, "cos_r_negg": [0.40041670203208923, 0.4457572102546692, 0.4869686961174011], "cos_apical_negg": [0.08335276693105698, 0.14289376139640808, 0.2409048080444336], "cos_Ac_negg": [0.4003782272338867, 0.4456818997859955, 0.486863911151886], "r_norm": [0.2247849553823471, 0.22020334005355835, 0.21887263655662537]}, {"step": 2800, "epoch": 5, "train_loss": 0.0953345000743866, "cos_r_negg": [0.38028910756111145, 0.4324725866317749, 0.47883039712905884], "cos_apical_negg": [0.09604420512914658, 0.1389172524213791, 0.1950380504131317], "cos_Ac_negg": [0.38023999333381653, 0.43236660957336426, 0.4787330627441406], "r_norm": [0.2579306364059448, 0.2504432499408722, 0.24913740158081055]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9644, "test_loss": 0.14248134422302247}, {"step": 2900, "epoch": 6, "train_loss": 0.01733783632516861, "cos_r_negg": [0.34765544533729553, 0.4037877917289734, 0.4658434987068176], "cos_apical_negg": [0.0725863128900528, 0.17288920283317566, 0.21283870935440063], "cos_Ac_negg": [0.34756898880004883, 0.4036415219306946, 0.46571725606918335], "r_norm": [0.2390182614326477, 0.23241755366325378, 0.2319326102733612]}, {"step": 3000, "epoch": 6, "train_loss": 0.23049652576446533, "cos_r_negg": [0.36030441522598267, 0.41825050115585327, 0.47125720977783203], "cos_apical_negg": [0.07580869644880295, 0.14458076655864716, 0.21424773335456848], "cos_Ac_negg": [0.3602030277252197, 0.4181329607963562, 0.4712256193161011], "r_norm": [0.232564777135849, 0.22713430225849152, 0.22508463263511658], "ldr": {"dL_sdil": -0.0049345046281814575, "dL_bp": -0.005487807095050812, "ratio": 0.8991760356576034}}, {"step": 3100, "epoch": 6, "train_loss": 0.06224461644887924, "cos_r_negg": [0.37403592467308044, 0.4602539837360382, 0.4656350612640381], "cos_apical_negg": [0.084104984998703, 0.18456366658210754, 0.23997732996940613], "cos_Ac_negg": [0.3739563226699829, 0.46012061834335327, 0.46556568145751953], "r_norm": [0.2350768744945526, 0.22893840074539185, 0.22728988528251648]}, {"step": 3200, "epoch": 6, "train_loss": 0.0449090451002121, "cos_r_negg": [0.35728710889816284, 0.41038230061531067, 0.4553821086883545], "cos_apical_negg": [0.07288661599159241, 0.1684381067752838, 0.22839972376823425], "cos_Ac_negg": [0.3572527766227722, 0.4102838635444641, 0.45527946949005127], "r_norm": [0.24285468459129333, 0.23665675520896912, 0.2350439578294754]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9672, "test_loss": 0.13613516082763671}, {"step": 3300, "epoch": 7, "train_loss": 0.029432538896799088, "cos_r_negg": [0.3691403269767761, 0.41885921359062195, 0.4724518060684204], "cos_apical_negg": [0.08806461095809937, 0.15996243059635162, 0.230827197432518], "cos_Ac_negg": [0.36912912130355835, 0.41879940032958984, 0.4724229574203491], "r_norm": [0.2313859462738037, 0.22590097784996033, 0.2239949256181717]}, {"step": 3400, "epoch": 7, "train_loss": 0.09313170611858368, "cos_r_negg": [0.3199690878391266, 0.3703828752040863, 0.393064022064209], "cos_apical_negg": [0.07493884861469269, 0.13648273050785065, 0.15313021838665009], "cos_Ac_negg": [0.3199384808540344, 0.37025684118270874, 0.39303058385849], "r_norm": [0.23286309838294983, 0.2264416515827179, 0.22547772526741028]}, {"step": 3500, "epoch": 7, "train_loss": 0.010211452841758728, "cos_r_negg": [0.3271755576133728, 0.3839573264122009, 0.40030205249786377], "cos_apical_negg": [0.06485243886709213, 0.13228990137577057, 0.17978887259960175], "cos_Ac_negg": [0.3271646499633789, 0.3838975727558136, 0.4002265930175781], "r_norm": [0.22983810305595398, 0.22434550523757935, 0.22287964820861816], "ldr": {"dL_sdil": -0.00641883909702301, "dL_bp": -0.007397115230560303, "ratio": 0.8677489665842083}}, {"step": 3600, "epoch": 7, "train_loss": 0.0180648360401392, "cos_r_negg": [0.3579770624637604, 0.3948354125022888, 0.43406233191490173], "cos_apical_negg": [0.06698261201381683, 0.1468813121318817, 0.25213533639907837], "cos_Ac_negg": [0.35793596506118774, 0.39472973346710205, 0.4339659810066223], "r_norm": [0.24397656321525574, 0.23807041347026825, 0.23590627312660217]}, {"step": 3700, "epoch": 7, "train_loss": 0.02788790501654148, "cos_r_negg": [0.35952135920524597, 0.3989863991737366, 0.44728466868400574], "cos_apical_negg": [0.07870279997587204, 0.14041903614997864, 0.25244852900505066], "cos_Ac_negg": [0.3594464659690857, 0.3988378942012787, 0.4472004771232605], "r_norm": [0.24892956018447876, 0.24216148257255554, 0.2400968372821808]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9686, "test_loss": 0.13276984481811524}, {"step": 3800, "epoch": 8, "train_loss": 0.02665073610842228, "cos_r_negg": [0.3220789134502411, 0.36207181215286255, 0.4071982502937317], "cos_apical_negg": [0.06650383025407791, 0.11593539267778397, 0.18757429718971252], "cos_Ac_negg": [0.32200467586517334, 0.3619110584259033, 0.4070832133293152], "r_norm": [0.2055143117904663, 0.20011088252067566, 0.19833213090896606]}, {"step": 3900, "epoch": 8, "train_loss": 0.13355880975723267, "cos_r_negg": [0.3140375316143036, 0.3537817597389221, 0.41636207699775696], "cos_apical_negg": [0.061907924711704254, 0.1273750215768814, 0.18623270094394684], "cos_Ac_negg": [0.31402575969696045, 0.3536485731601715, 0.4162844121456146], "r_norm": [0.22917672991752625, 0.22195549309253693, 0.22086377441883087]}, {"step": 4000, "epoch": 8, "train_loss": 0.04349532723426819, "cos_r_negg": [0.31868577003479004, 0.3747909367084503, 0.4304412603378296], "cos_apical_negg": [0.08351548761129379, 0.11740659177303314, 0.20919513702392578], "cos_Ac_negg": [0.31867071986198425, 0.37468305230140686, 0.43036019802093506], "r_norm": [0.20098544657230377, 0.19529084861278534, 0.19426798820495605], "ldr": {"dL_sdil": -0.004782721400260925, "dL_bp": -0.005733594298362732, "ratio": 0.8341576245857969}}, {"step": 4100, "epoch": 8, "train_loss": 0.012566429562866688, "cos_r_negg": [0.3185289800167084, 0.35179272294044495, 0.3757590651512146], "cos_apical_negg": [0.059507105499506, 0.11047831177711487, 0.15767443180084229], "cos_Ac_negg": [0.31850719451904297, 0.35168856382369995, 0.3757178485393524], "r_norm": [0.2059113085269928, 0.20031198859214783, 0.198930025100708]}, {"step": 4200, "epoch": 8, "train_loss": 0.0665077418088913, "cos_r_negg": [0.2978224456310272, 0.33452531695365906, 0.3787321448326111], "cos_apical_negg": [0.06658408045768738, 0.12088974565267563, 0.17964455485343933], "cos_Ac_negg": [0.2977486848831177, 0.3344022035598755, 0.3787045180797577], "r_norm": [0.20869377255439758, 0.20384074747562408, 0.20140281319618225]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9683, "test_loss": 0.14483663024902343}, {"step": 4300, "epoch": 9, "train_loss": 0.03931870311498642, "cos_r_negg": [0.2826802730560303, 0.33147820830345154, 0.4041232168674469], "cos_apical_negg": [0.06308436393737793, 0.11510062217712402, 0.15491870045661926], "cos_Ac_negg": [0.28264129161834717, 0.33136460185050964, 0.4040958881378174], "r_norm": [0.2176712155342102, 0.21093179285526276, 0.20935454964637756]}, {"step": 4400, "epoch": 9, "train_loss": 0.023374365642666817, "cos_r_negg": [0.3270260691642761, 0.3662740886211395, 0.4073263108730316], "cos_apical_negg": [0.07388993352651596, 0.13460934162139893, 0.1963604986667633], "cos_Ac_negg": [0.32702016830444336, 0.3661465346813202, 0.40726691484451294], "r_norm": [0.22733575105667114, 0.2197837233543396, 0.21842718124389648]}, {"step": 4500, "epoch": 9, "train_loss": 0.12062329053878784, "cos_r_negg": [0.3130398392677307, 0.35118821263313293, 0.40689733624458313], "cos_apical_negg": [0.07170061022043228, 0.13534925878047943, 0.22704070806503296], "cos_Ac_negg": [0.31298938393592834, 0.35103684663772583, 0.4067959785461426], "r_norm": [0.2059287428855896, 0.20092462003231049, 0.19870661199092865], "ldr": {"dL_sdil": -0.005284205079078674, "dL_bp": -0.00652487576007843, "ratio": 0.8098552789938727}}, {"step": 4600, "epoch": 9, "train_loss": 0.024062493816018105, "cos_r_negg": [0.27017152309417725, 0.3290991187095642, 0.35464465618133545], "cos_apical_negg": [0.06068180501461029, 0.10684116184711456, 0.1582408845424652], "cos_Ac_negg": [0.27016592025756836, 0.3290368616580963, 0.3546128571033478], "r_norm": [0.2156028002500534, 0.20901551842689514, 0.20716522634029388]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9695, "test_loss": 0.14785444831848144}, {"step": 4700, "epoch": 10, "train_loss": 0.008161413483321667, "cos_r_negg": [0.2580244243144989, 0.29799771308898926, 0.33508923649787903], "cos_apical_negg": [0.05917653441429138, 0.10824452340602875, 0.14844365417957306], "cos_Ac_negg": [0.2579764723777771, 0.2978969216346741, 0.3350910246372223], "r_norm": [0.20591787993907928, 0.19990742206573486, 0.1984596848487854]}, {"step": 4800, "epoch": 10, "train_loss": 0.0478949099779129, "cos_r_negg": [0.2587451934814453, 0.31345027685165405, 0.3682030439376831], "cos_apical_negg": [0.06689292192459106, 0.12296228855848312, 0.15581010282039642], "cos_Ac_negg": [0.25869759917259216, 0.3133394420146942, 0.3681696057319641], "r_norm": [0.19997137784957886, 0.1951225996017456, 0.19302034378051758]}, {"step": 4900, "epoch": 10, "train_loss": 0.016231359913945198, "cos_r_negg": [0.2460336834192276, 0.28084877133369446, 0.32604533433914185], "cos_apical_negg": [0.06351970136165619, 0.13150522112846375, 0.12353996932506561], "cos_Ac_negg": [0.2459530234336853, 0.2806828022003174, 0.3260475993156433], "r_norm": [0.23257997632026672, 0.2268572747707367, 0.22457227110862732]}, {"step": 5000, "epoch": 10, "train_loss": 0.006949783768504858, "cos_r_negg": [0.2522090971469879, 0.29522794485092163, 0.33196166157722473], "cos_apical_negg": [0.06330932676792145, 0.11816816031932831, 0.17218172550201416], "cos_Ac_negg": [0.25216856598854065, 0.29511621594429016, 0.33194196224212646], "r_norm": [0.1951100081205368, 0.188955619931221, 0.1878432333469391], "ldr": {"dL_sdil": -0.005989968776702881, "dL_bp": -0.007699951529502869, "ratio": 0.7779229198718879}}, {"step": 5100, "epoch": 10, "train_loss": 0.04045986384153366, "cos_r_negg": [0.27094173431396484, 0.32451125979423523, 0.3775748312473297], "cos_apical_negg": [0.05169527232646942, 0.09858563542366028, 0.18976131081581116], "cos_Ac_negg": [0.2708813548088074, 0.3243916928768158, 0.37751856446266174], "r_norm": [0.1989247053861618, 0.19375018775463104, 0.19190067052841187]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9693, "test_loss": 0.14779638671875}, {"step": 5200, "epoch": 11, "train_loss": 0.002651508431881666, "cos_r_negg": [0.2496313452720642, 0.28159278631210327, 0.35537096858024597], "cos_apical_negg": [0.04971758648753166, 0.09500518441200256, 0.1748282015323639], "cos_Ac_negg": [0.24960198998451233, 0.2814895510673523, 0.355343759059906], "r_norm": [0.19997933506965637, 0.19511821866035461, 0.19288817048072815]}, {"step": 5300, "epoch": 11, "train_loss": 0.03633364289999008, "cos_r_negg": [0.2568783164024353, 0.29175281524658203, 0.350791871547699], "cos_apical_negg": [0.0692330151796341, 0.09364697337150574, 0.19000670313835144], "cos_Ac_negg": [0.25686150789260864, 0.29163771867752075, 0.35071372985839844], "r_norm": [0.22654028236865997, 0.21995390951633453, 0.2184176743030548]}, {"step": 5400, "epoch": 11, "train_loss": 0.033560846000909805, "cos_r_negg": [0.23786437511444092, 0.2767171561717987, 0.3369234800338745], "cos_apical_negg": [0.050773985683918, 0.09891113638877869, 0.1714353859424591], "cos_Ac_negg": [0.23784521222114563, 0.27660128474235535, 0.33687642216682434], "r_norm": [0.2021322101354599, 0.1962040662765503, 0.19472111761569977]}, {"step": 5500, "epoch": 11, "train_loss": 0.04395921155810356, "cos_r_negg": [0.21185636520385742, 0.26318007707595825, 0.3347349762916565], "cos_apical_negg": [0.08126802742481232, 0.0925680547952652, 0.16087763011455536], "cos_Ac_negg": [0.21180152893066406, 0.26308244466781616, 0.3347211182117462], "r_norm": [0.21441692113876343, 0.20836228132247925, 0.2062130868434906], "ldr": {"dL_sdil": -0.005746632814407349, "dL_bp": -0.008455976843833923, "ratio": 0.6795941995273768}}, {"step": 5600, "epoch": 11, "train_loss": 0.015303989872336388, "cos_r_negg": [0.22986271977424622, 0.2702818512916565, 0.3128160536289215], "cos_apical_negg": [0.050982698798179626, 0.09545817971229553, 0.13293741643428802], "cos_Ac_negg": [0.2298513650894165, 0.27018457651138306, 0.3127765655517578], "r_norm": [0.20144091546535492, 0.19484953582286835, 0.19381265342235565]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9699, "test_loss": 0.15268256072998046}, {"step": 5700, "epoch": 12, "train_loss": 0.003603791818022728, "cos_r_negg": [0.22636467218399048, 0.2707720994949341, 0.32874518632888794], "cos_apical_negg": [0.04032183811068535, 0.096725694835186, 0.14258968830108643], "cos_Ac_negg": [0.22631829977035522, 0.2706739902496338, 0.3286795914173126], "r_norm": [0.19349460303783417, 0.18741345405578613, 0.18626046180725098]}, {"step": 5800, "epoch": 12, "train_loss": 0.020854540169239044, "cos_r_negg": [0.21486562490463257, 0.2630091905593872, 0.3170546293258667], "cos_apical_negg": [0.06276147067546844, 0.09671137481927872, 0.15367133915424347], "cos_Ac_negg": [0.21481385827064514, 0.26289957761764526, 0.3170144557952881], "r_norm": [0.19152942299842834, 0.18530118465423584, 0.18406842648983002]}, {"step": 5900, "epoch": 12, "train_loss": 0.008579861372709274, "cos_r_negg": [0.23036068677902222, 0.268257200717926, 0.32172608375549316], "cos_apical_negg": [0.04922834411263466, 0.09663659334182739, 0.1473350077867508], "cos_Ac_negg": [0.23033815622329712, 0.2681354880332947, 0.32169950008392334], "r_norm": [0.19517508149147034, 0.18891669809818268, 0.18755337595939636]}, {"step": 6000, "epoch": 12, "train_loss": 0.023891856893897057, "cos_r_negg": [0.21371719241142273, 0.24589970707893372, 0.3007495701313019], "cos_apical_negg": [0.05639013648033142, 0.09345866739749908, 0.1516299694776535], "cos_Ac_negg": [0.21364545822143555, 0.24576793611049652, 0.30071553587913513], "r_norm": [0.19958920776844025, 0.19289015233516693, 0.19160467386245728], "ldr": {"dL_sdil": -0.005817905068397522, "dL_bp": -0.010013014078140259, "ratio": 0.5810343442039877}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9689, "test_loss": 0.15575114250183106}, {"step": 6100, "epoch": 13, "train_loss": 0.031186243519186974, "cos_r_negg": [0.23160181939601898, 0.2642801105976105, 0.32137686014175415], "cos_apical_negg": [0.06791743636131287, 0.10517482459545135, 0.14122797548770905], "cos_Ac_negg": [0.23152336478233337, 0.2641903758049011, 0.3214069604873657], "r_norm": [0.2154746651649475, 0.2092437744140625, 0.20703613758087158]}, {"step": 6200, "epoch": 13, "train_loss": 0.006884851958602667, "cos_r_negg": [0.2182973176240921, 0.24919408559799194, 0.3115878403186798], "cos_apical_negg": [0.0568549670279026, 0.0780751183629036, 0.12147295475006104], "cos_Ac_negg": [0.2182532548904419, 0.24912229180335999, 0.3115708827972412], "r_norm": [0.19051426649093628, 0.18517754971981049, 0.18321485817432404]}, {"step": 6300, "epoch": 13, "train_loss": 0.03043420985341072, "cos_r_negg": [0.20692133903503418, 0.2380739152431488, 0.2872515916824341], "cos_apical_negg": [0.056095898151397705, 0.09552943706512451, 0.12139946967363358], "cos_Ac_negg": [0.2069166749715805, 0.23799897730350494, 0.2872368395328522], "r_norm": [0.1792515069246292, 0.17412413656711578, 0.17261335253715515]}, {"step": 6400, "epoch": 13, "train_loss": 0.018544811755418777, "cos_r_negg": [0.2081642597913742, 0.24410881102085114, 0.30179691314697266], "cos_apical_negg": [0.05687960609793663, 0.09739081561565399, 0.14163720607757568], "cos_Ac_negg": [0.20812173187732697, 0.24404558539390564, 0.301847904920578], "r_norm": [0.20159514248371124, 0.19521547853946686, 0.19361643493175507]}, {"step": 6500, "epoch": 13, "train_loss": 0.00039314196328632534, "cos_r_negg": [0.1892247200012207, 0.2203974425792694, 0.27506309747695923], "cos_apical_negg": [0.04459456354379654, 0.07980211079120636, 0.13723260164260864], "cos_Ac_negg": [0.1891787201166153, 0.22031477093696594, 0.2750839591026306], "r_norm": [0.18581117689609528, 0.17956629395484924, 0.17843253910541534], "ldr": {"dL_sdil": -0.005495429039001465, "dL_bp": -0.008063137531280518, "ratio": 0.6815497091153707}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9692, "test_loss": 0.16447142143249513}, {"step": 6600, "epoch": 14, "train_loss": 0.0031038797460496426, "cos_r_negg": [0.19721244275569916, 0.23586884140968323, 0.298753559589386], "cos_apical_negg": [0.05631585419178009, 0.08089663088321686, 0.14550811052322388], "cos_Ac_negg": [0.19716256856918335, 0.23576107621192932, 0.29876089096069336], "r_norm": [0.18716417253017426, 0.18181508779525757, 0.18002164363861084]}, {"step": 6700, "epoch": 14, "train_loss": 0.005571616813540459, "cos_r_negg": [0.19782164692878723, 0.23288075625896454, 0.2836465537548065], "cos_apical_negg": [0.047006845474243164, 0.08577421307563782, 0.14684957265853882], "cos_Ac_negg": [0.19777968525886536, 0.23278269171714783, 0.2836758494377136], "r_norm": [0.19478219747543335, 0.18811435997486115, 0.18674880266189575]}, {"step": 6800, "epoch": 14, "train_loss": 0.024110812693834305, "cos_r_negg": [0.18011140823364258, 0.21659880876541138, 0.26940786838531494], "cos_apical_negg": [0.04869562387466431, 0.08290772140026093, 0.12963669002056122], "cos_Ac_negg": [0.18003004789352417, 0.21650764346122742, 0.2693968415260315], "r_norm": [0.193019300699234, 0.18687675893306732, 0.18526864051818848]}, {"step": 6900, "epoch": 14, "train_loss": 0.024755142629146576, "cos_r_negg": [0.16172689199447632, 0.19865772128105164, 0.26105427742004395], "cos_apical_negg": [0.0629316121339798, 0.08489827811717987, 0.11592637002468109], "cos_Ac_negg": [0.1616540253162384, 0.19857871532440186, 0.2610599994659424], "r_norm": [0.19401678442955017, 0.18787765502929688, 0.18621554970741272]}, {"step": 7000, "epoch": 14, "train_loss": 0.02067204751074314, "cos_r_negg": [0.1930568516254425, 0.23053166270256042, 0.29817941784858704], "cos_apical_negg": [0.05957508087158203, 0.08531548082828522, 0.16373056173324585], "cos_Ac_negg": [0.19296862185001373, 0.2304086685180664, 0.2981274724006653], "r_norm": [0.18739569187164307, 0.1816098988056183, 0.18000106513500214], "ldr": {"dL_sdil": -0.005046382546424866, "dL_bp": -0.00739249587059021, "ratio": 0.682635828922278}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.969, "test_loss": 0.17767958984375}], "final": {"test_acc": 0.969, "test_loss": 0.17767958984375, "wall_s": 42.838538646698, "cos_r_negg": [0.18309010565280914, 0.2141142189502716, 0.2727559804916382], "cos_apical_negg": [0.06834009289741516, 0.08814489841461182, 0.13101357221603394], "cos_Ac_negg": [0.18302752077579498, 0.21398913860321045, 0.27272993326187134]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p02_res1_s0.json b/results/diag_nuis_low_rho0p02_res1_s0.json new file mode 100644 index 0000000..01ccf29 --- /dev/null +++ b/results/diag_nuis_low_rho0p02_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.02, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p02_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.019516076892614365, -0.02552611567080021, 0.03376470506191254], "cos_apical_negg": [-0.019525274634361267, -0.025608714669942856, 0.033514510840177536], "cos_Ac_negg": [-0.019516058266162872, -0.02552611753344536, 0.03376470133662224], "r_norm": [4.898172378540039, 4.826802730560303, 4.823856353759766], "ldr": {"dL_sdil": -0.009198188781738281, "dL_bp": -0.2406764030456543, "ratio": 0.03821807483134714}}, {"step": 100, "epoch": 0, "train_loss": 0.20666691660881042, "cos_r_negg": [0.45621293783187866, 0.3815845847129822, 0.3604835271835327], "cos_apical_negg": [0.2548171281814575, 0.2229165881872177, 0.2502298951148987], "cos_Ac_negg": [0.4563215970993042, 0.38170552253723145, 0.3605661392211914], "r_norm": [0.8244898319244385, 0.8078850507736206, 0.8057104349136353]}, {"step": 200, "epoch": 0, "train_loss": 0.3330998122692108, "cos_r_negg": [0.47436001896858215, 0.4069191813468933, 0.3816928565502167], "cos_apical_negg": [0.24737149477005005, 0.22371184825897217, 0.2461678683757782], "cos_Ac_negg": [0.47450393438339233, 0.4071381986141205, 0.3819814622402191], "r_norm": [0.7478988766670227, 0.730629563331604, 0.72956383228302]}, {"step": 300, "epoch": 0, "train_loss": 0.17292667925357819, "cos_r_negg": [0.500956654548645, 0.47095879912376404, 0.42712223529815674], "cos_apical_negg": [0.2329605221748352, 0.2119992971420288, 0.23979687690734863], "cos_Ac_negg": [0.50104820728302, 0.4710274934768677, 0.4271710515022278], "r_norm": [0.5720133781433105, 0.5613110065460205, 0.5610545873641968]}, {"step": 400, "epoch": 0, "train_loss": 0.12120312452316284, "cos_r_negg": [0.5187759399414062, 0.48503759503364563, 0.4563548266887665], "cos_apical_negg": [0.2260979413986206, 0.2193305790424347, 0.24290788173675537], "cos_Ac_negg": [0.5188868045806885, 0.48505544662475586, 0.4563472270965576], "r_norm": [0.5730111002922058, 0.5584796667098999, 0.5582184195518494]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9455, "test_loss": 0.1858793228149414}, {"step": 500, "epoch": 1, "train_loss": 0.22908438742160797, "cos_r_negg": [0.5118932723999023, 0.4822896122932434, 0.47202080488204956], "cos_apical_negg": [0.19514621794223785, 0.18528658151626587, 0.22929653525352478], "cos_Ac_negg": [0.5121138691902161, 0.4823378920555115, 0.47203999757766724], "r_norm": [0.4961223006248474, 0.478294312953949, 0.4794073700904846], "ldr": {"dL_sdil": -0.012139663100242615, "dL_bp": -0.010848388075828552, "ratio": 1.1190292064948497}}, {"step": 600, "epoch": 1, "train_loss": 0.09079079329967499, "cos_r_negg": [0.5184227228164673, 0.5019474029541016, 0.4570058584213257], "cos_apical_negg": [0.1823062300682068, 0.19016176462173462, 0.2272079885005951], "cos_Ac_negg": [0.518621563911438, 0.5019934177398682, 0.45703262090682983], "r_norm": [0.43909209966659546, 0.42774102091789246, 0.4260190725326538]}, {"step": 700, "epoch": 1, "train_loss": 0.21720203757286072, "cos_r_negg": [0.5171608924865723, 0.5006321668624878, 0.468919575214386], "cos_apical_negg": [0.16877971589565277, 0.17867717146873474, 0.22401881217956543], "cos_Ac_negg": [0.5175011157989502, 0.5006343126296997, 0.4689297080039978], "r_norm": [0.4626578092575073, 0.44650548696517944, 0.4488585889339447]}, {"step": 800, "epoch": 1, "train_loss": 0.13265103101730347, "cos_r_negg": [0.5302672386169434, 0.5384685397148132, 0.4882376790046692], "cos_apical_negg": [0.15258991718292236, 0.17883214354515076, 0.21039175987243652], "cos_Ac_negg": [0.5305817127227783, 0.5384242534637451, 0.48820960521698], "r_norm": [0.37950968742370605, 0.3676168918609619, 0.36676251888275146]}, {"step": 900, "epoch": 1, "train_loss": 0.20370136201381683, "cos_r_negg": [0.5451664924621582, 0.5469822883605957, 0.47349900007247925], "cos_apical_negg": [0.16239389777183533, 0.18302062153816223, 0.21378210186958313], "cos_Ac_negg": [0.5452289581298828, 0.5469443202018738, 0.47342297434806824], "r_norm": [0.37063032388687134, 0.3622225522994995, 0.36039620637893677]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9529, "test_loss": 0.15865937118530274}, {"step": 1000, "epoch": 2, "train_loss": 0.029328424483537674, "cos_r_negg": [0.5289598703384399, 0.5364674925804138, 0.4905230700969696], "cos_apical_negg": [0.13371966779232025, 0.1510743498802185, 0.19319716095924377], "cos_Ac_negg": [0.5291118025779724, 0.5364547967910767, 0.4904087781906128], "r_norm": [0.3019307851791382, 0.2927197813987732, 0.29203489422798157], "ldr": {"dL_sdil": -0.006969355046749115, "dL_bp": -0.005919985473155975, "ratio": 1.177258808885846}}, {"step": 1100, "epoch": 2, "train_loss": 0.10531969368457794, "cos_r_negg": [0.5412123203277588, 0.5536156892776489, 0.5019960403442383], "cos_apical_negg": [0.13393628597259521, 0.15139706432819366, 0.20108921825885773], "cos_Ac_negg": [0.5413720607757568, 0.5536148548126221, 0.5018433928489685], "r_norm": [0.37149864435195923, 0.3579740822315216, 0.3603286147117615]}, {"step": 1200, "epoch": 2, "train_loss": 0.18178343772888184, "cos_r_negg": [0.5275225639343262, 0.5536405444145203, 0.5052825212478638], "cos_apical_negg": [0.1327957957983017, 0.14455966651439667, 0.18889185786247253], "cos_Ac_negg": [0.527618408203125, 0.5536795854568481, 0.5051732063293457], "r_norm": [0.2758646309375763, 0.26887577772140503, 0.2693221867084503]}, {"step": 1300, "epoch": 2, "train_loss": 0.13751375675201416, "cos_r_negg": [0.5319006443023682, 0.5425025820732117, 0.48974841833114624], "cos_apical_negg": [0.12245390564203262, 0.14747200906276703, 0.19146475195884705], "cos_Ac_negg": [0.5320289134979248, 0.5424505472183228, 0.4895438253879547], "r_norm": [0.2822554111480713, 0.27503806352615356, 0.27348774671554565]}, {"step": 1400, "epoch": 2, "train_loss": 0.1919175386428833, "cos_r_negg": [0.4942934811115265, 0.5211362242698669, 0.469143271446228], "cos_apical_negg": [0.13863955438137054, 0.15377335250377655, 0.1846141815185547], "cos_Ac_negg": [0.49437832832336426, 0.520835280418396, 0.4687177538871765], "r_norm": [0.2955261170864105, 0.28647202253341675, 0.2861825227737427]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9637, "test_loss": 0.12490827331542968}, {"step": 1500, "epoch": 3, "train_loss": 0.09113158285617828, "cos_r_negg": [0.49388402700424194, 0.5273165702819824, 0.4769553244113922], "cos_apical_negg": [0.10814551264047623, 0.13171860575675964, 0.1783793568611145], "cos_Ac_negg": [0.49401742219924927, 0.5272470116615295, 0.4765438735485077], "r_norm": [0.2476242333650589, 0.24017561972141266, 0.24017147719860077], "ldr": {"dL_sdil": -0.0050140321254730225, "dL_bp": -0.0049041807651519775, "ratio": 1.0223995332924154}}, {"step": 1600, "epoch": 3, "train_loss": 0.09694245457649231, "cos_r_negg": [0.503745436668396, 0.5409880876541138, 0.5105512142181396], "cos_apical_negg": [0.12057998776435852, 0.13434819877147675, 0.18615035712718964], "cos_Ac_negg": [0.5037381052970886, 0.5408003330230713, 0.5100628137588501], "r_norm": [0.30542612075805664, 0.295025497674942, 0.2949987053871155]}, {"step": 1700, "epoch": 3, "train_loss": 0.042790692299604416, "cos_r_negg": [0.4831027686595917, 0.5262888073921204, 0.4767081141471863], "cos_apical_negg": [0.12841689586639404, 0.14515358209609985, 0.19567400217056274], "cos_Ac_negg": [0.4830111265182495, 0.5262072086334229, 0.4762493669986725], "r_norm": [0.2926684617996216, 0.2838863134384155, 0.2848629951477051]}, {"step": 1800, "epoch": 3, "train_loss": 0.025837283581495285, "cos_r_negg": [0.49439430236816406, 0.5511816740036011, 0.48297521471977234], "cos_apical_negg": [0.08975066989660263, 0.12552043795585632, 0.1713440716266632], "cos_Ac_negg": [0.4943971037864685, 0.5510495901107788, 0.48242872953414917], "r_norm": [0.2425144612789154, 0.23646119236946106, 0.23406532406806946]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9674, "test_loss": 0.11496790046691895}, {"step": 1900, "epoch": 4, "train_loss": 0.04518939554691315, "cos_r_negg": [0.48827606439590454, 0.5391676425933838, 0.4935782551765442], "cos_apical_negg": [0.11472448706626892, 0.1402273178100586, 0.1942419409751892], "cos_Ac_negg": [0.48823282122612, 0.5389991998672485, 0.49292588233947754], "r_norm": [0.3024155795574188, 0.2952362298965454, 0.2935969829559326]}, {"step": 2000, "epoch": 4, "train_loss": 0.09094434231519699, "cos_r_negg": [0.4285632073879242, 0.48903414607048035, 0.4430459141731262], "cos_apical_negg": [0.0712447464466095, 0.10120940208435059, 0.15749140083789825], "cos_Ac_negg": [0.4285363256931305, 0.4889431893825531, 0.4423690438270569], "r_norm": [0.19939634203910828, 0.19333955645561218, 0.19306454062461853], "ldr": {"dL_sdil": -0.004258647561073303, "dL_bp": -0.004662469029426575, "ratio": 0.9133889220915776}}, {"step": 2100, "epoch": 4, "train_loss": 0.04867567867040634, "cos_r_negg": [0.4485446512699127, 0.5176587104797363, 0.4661404490470886], "cos_apical_negg": [0.09516894072294235, 0.1227864995598793, 0.18243567645549774], "cos_Ac_negg": [0.4484497308731079, 0.5175020098686218, 0.4654655158519745], "r_norm": [0.2640710473060608, 0.2580794394016266, 0.25592613220214844]}, {"step": 2200, "epoch": 4, "train_loss": 0.005549731198698282, "cos_r_negg": [0.4341243505477905, 0.49059516191482544, 0.4355831444263458], "cos_apical_negg": [0.08100329339504242, 0.10584701597690582, 0.1655983179807663], "cos_Ac_negg": [0.43403735756874084, 0.49035704135894775, 0.43483299016952515], "r_norm": [0.25272154808044434, 0.24526101350784302, 0.24376076459884644]}, {"step": 2300, "epoch": 4, "train_loss": 0.11908858269453049, "cos_r_negg": [0.41476237773895264, 0.4658155143260956, 0.42012280225753784], "cos_apical_negg": [0.08445233106613159, 0.11838938295841217, 0.16716186702251434], "cos_Ac_negg": [0.4147198796272278, 0.46558678150177, 0.41937845945358276], "r_norm": [0.21195140480995178, 0.20710352063179016, 0.20612670481204987]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9706, "test_loss": 0.11300087585449219}, {"step": 2400, "epoch": 5, "train_loss": 0.09114673733711243, "cos_r_negg": [0.41651755571365356, 0.47811898589134216, 0.4303111433982849], "cos_apical_negg": [0.07098812609910965, 0.09009087085723877, 0.16010312736034393], "cos_Ac_negg": [0.41635245084762573, 0.4779331684112549, 0.4294087290763855], "r_norm": [0.20994238555431366, 0.2038857340812683, 0.20224055647850037]}, {"step": 2500, "epoch": 5, "train_loss": 0.0506490059196949, "cos_r_negg": [0.38574618101119995, 0.4409067928791046, 0.397209107875824], "cos_apical_negg": [0.07910504192113876, 0.08861284703016281, 0.16518300771713257], "cos_Ac_negg": [0.38568058609962463, 0.44085222482681274, 0.3965015113353729], "r_norm": [0.21368306875228882, 0.20854078233242035, 0.20763856172561646], "ldr": {"dL_sdil": -0.00640907883644104, "dL_bp": -0.007272765040397644, "ratio": 0.8812437636635954}}, {"step": 2600, "epoch": 5, "train_loss": 0.049843236804008484, "cos_r_negg": [0.39549702405929565, 0.44585907459259033, 0.403192937374115], "cos_apical_negg": [0.07233475893735886, 0.09070632606744766, 0.16163372993469238], "cos_Ac_negg": [0.395357221364975, 0.44567298889160156, 0.40239015221595764], "r_norm": [0.19365227222442627, 0.18826515972614288, 0.18756335973739624]}, {"step": 2700, "epoch": 5, "train_loss": 0.011957159265875816, "cos_r_negg": [0.41444921493530273, 0.4680020213127136, 0.42595210671424866], "cos_apical_negg": [0.06405209004878998, 0.09268269687891006, 0.15636903047561646], "cos_Ac_negg": [0.41426366567611694, 0.46777641773223877, 0.42501968145370483], "r_norm": [0.1665506362915039, 0.16178865730762482, 0.16212913393974304]}, {"step": 2800, "epoch": 5, "train_loss": 0.0497964546084404, "cos_r_negg": [0.3917809724807739, 0.45230811834335327, 0.41274556517601013], "cos_apical_negg": [0.07781335711479187, 0.1003563329577446, 0.17138859629631042], "cos_Ac_negg": [0.3917839825153351, 0.45210278034210205, 0.4119354784488678], "r_norm": [0.20583036541938782, 0.20006787776947021, 0.19989202916622162]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9697, "test_loss": 0.11425394191741943}, {"step": 2900, "epoch": 6, "train_loss": 0.016345564275979996, "cos_r_negg": [0.3668573498725891, 0.4212730824947357, 0.3867816627025604], "cos_apical_negg": [0.05176198109984398, 0.09197431057691574, 0.1654084324836731], "cos_Ac_negg": [0.3670150935649872, 0.42115360498428345, 0.3859657645225525], "r_norm": [0.18884789943695068, 0.1829959899187088, 0.18494556844234467]}, {"step": 3000, "epoch": 6, "train_loss": 0.06740796566009521, "cos_r_negg": [0.35739368200302124, 0.41632911562919617, 0.3758406341075897], "cos_apical_negg": [0.04749305546283722, 0.07950197160243988, 0.1474684327840805], "cos_Ac_negg": [0.35742366313934326, 0.41612905263900757, 0.37502217292785645], "r_norm": [0.1672361195087433, 0.16116869449615479, 0.1613539755344391], "ldr": {"dL_sdil": -0.004773199558258057, "dL_bp": -0.006503865122795105, "ratio": 0.7339019902993812}}, {"step": 3100, "epoch": 6, "train_loss": 0.015779832378029823, "cos_r_negg": [0.33921587467193604, 0.401305615901947, 0.3627150058746338], "cos_apical_negg": [0.04513528570532799, 0.08053319901227951, 0.14738032221794128], "cos_Ac_negg": [0.33924078941345215, 0.40108269453048706, 0.36195629835128784], "r_norm": [0.19078296422958374, 0.1855381280183792, 0.18521305918693542]}, {"step": 3200, "epoch": 6, "train_loss": 0.014715760946273804, "cos_r_negg": [0.323300838470459, 0.3751212954521179, 0.3425447344779968], "cos_apical_negg": [0.04321293905377388, 0.07752640545368195, 0.14486974477767944], "cos_Ac_negg": [0.3232835829257965, 0.37487027049064636, 0.34174519777297974], "r_norm": [0.15650531649589539, 0.15346120297908783, 0.15262295305728912]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9684, "test_loss": 0.13092685871124268}, {"step": 3300, "epoch": 7, "train_loss": 0.010674525052309036, "cos_r_negg": [0.30907344818115234, 0.35837921500205994, 0.32657375931739807], "cos_apical_negg": [0.05170029401779175, 0.08447684347629547, 0.1447277069091797], "cos_Ac_negg": [0.3089962899684906, 0.3581080436706543, 0.32574689388275146], "r_norm": [0.18258851766586304, 0.1787324845790863, 0.17720270156860352]}, {"step": 3400, "epoch": 7, "train_loss": 0.033174771815538406, "cos_r_negg": [0.28079208731651306, 0.32952362298965454, 0.2979621887207031], "cos_apical_negg": [0.04452217370271683, 0.08559951931238174, 0.1455911248922348], "cos_Ac_negg": [0.2807585299015045, 0.3293297290802002, 0.29722511768341064], "r_norm": [0.1638374626636505, 0.1596226990222931, 0.15870176255702972]}, {"step": 3500, "epoch": 7, "train_loss": 0.001852021785452962, "cos_r_negg": [0.2659572958946228, 0.3107095956802368, 0.28311192989349365], "cos_apical_negg": [0.031097523868083954, 0.07743818312883377, 0.14182418584823608], "cos_Ac_negg": [0.26604214310646057, 0.31052032113075256, 0.28239208459854126], "r_norm": [0.1802939772605896, 0.17595365643501282, 0.17571572959423065], "ldr": {"dL_sdil": -0.006213486194610596, "dL_bp": -0.00846630334854126, "ratio": 0.7339078153490893}}, {"step": 3600, "epoch": 7, "train_loss": 0.011132667772471905, "cos_r_negg": [0.29550132155418396, 0.34680211544036865, 0.3192564845085144], "cos_apical_negg": [0.052166856825351715, 0.08566244691610336, 0.1426910012960434], "cos_Ac_negg": [0.2954166531562805, 0.34657326340675354, 0.3185856342315674], "r_norm": [0.216933012008667, 0.21131376922130585, 0.21031340956687927]}, {"step": 3700, "epoch": 7, "train_loss": 0.009207136929035187, "cos_r_negg": [0.2779083549976349, 0.3308279514312744, 0.30681318044662476], "cos_apical_negg": [0.04207058250904083, 0.0821167528629303, 0.13696354627609253], "cos_Ac_negg": [0.27785173058509827, 0.33055371046066284, 0.306132972240448], "r_norm": [0.18489482998847961, 0.18016967177391052, 0.1788412630558014]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9723, "test_loss": 0.12359478569030762}, {"step": 3800, "epoch": 8, "train_loss": 0.002723423996940255, "cos_r_negg": [0.2539207339286804, 0.3019651174545288, 0.2805212438106537], "cos_apical_negg": [0.023168187588453293, 0.07803573459386826, 0.1375126987695694], "cos_Ac_negg": [0.25397467613220215, 0.30177628993988037, 0.2798832654953003], "r_norm": [0.17767879366874695, 0.17325571179389954, 0.1724102795124054]}, {"step": 3900, "epoch": 8, "train_loss": 0.018768351525068283, "cos_r_negg": [0.24070538580417633, 0.2874033451080322, 0.2684459090232849], "cos_apical_negg": [0.029751447960734367, 0.07550971955060959, 0.1321178376674652], "cos_Ac_negg": [0.2406652569770813, 0.2872641682624817, 0.26789575815200806], "r_norm": [0.1703643500804901, 0.1662348061800003, 0.1656225025653839]}, {"step": 4000, "epoch": 8, "train_loss": 0.013924628496170044, "cos_r_negg": [0.2321178913116455, 0.2789957523345947, 0.2601158022880554], "cos_apical_negg": [0.02238772250711918, 0.07014966011047363, 0.12876828014850616], "cos_Ac_negg": [0.23211203515529633, 0.2788405418395996, 0.25953006744384766], "r_norm": [0.15361139178276062, 0.15028171241283417, 0.14933691918849945], "ldr": {"dL_sdil": -0.005793191492557526, "dL_bp": -0.009559974074363708, "ratio": 0.6059840170584467}}, {"step": 4100, "epoch": 8, "train_loss": 0.016691818833351135, "cos_r_negg": [0.2363642156124115, 0.2781643271446228, 0.26152127981185913], "cos_apical_negg": [0.0323525071144104, 0.06793119758367538, 0.1332402378320694], "cos_Ac_negg": [0.236360564827919, 0.2779131531715393, 0.26090848445892334], "r_norm": [0.14785313606262207, 0.1430557668209076, 0.1428561955690384]}, {"step": 4200, "epoch": 8, "train_loss": 0.007424125913530588, "cos_r_negg": [0.22033420205116272, 0.2601127326488495, 0.24122940003871918], "cos_apical_negg": [0.03079233691096306, 0.07190176844596863, 0.1316319853067398], "cos_Ac_negg": [0.22028687596321106, 0.2598786950111389, 0.24067524075508118], "r_norm": [0.16593816876411438, 0.16193701326847076, 0.16203181445598602]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9722, "test_loss": 0.13240455589294434}, {"step": 4300, "epoch": 9, "train_loss": 0.006326611153781414, "cos_r_negg": [0.2145157754421234, 0.2527114450931549, 0.23701094090938568], "cos_apical_negg": [0.033963993191719055, 0.06959258019924164, 0.13042381405830383], "cos_Ac_negg": [0.2144797146320343, 0.2525511085987091, 0.23643437027931213], "r_norm": [0.18063950538635254, 0.17672720551490784, 0.175909161567688]}, {"step": 4400, "epoch": 9, "train_loss": 0.0011063176207244396, "cos_r_negg": [0.20485961437225342, 0.24688483774662018, 0.2341856062412262], "cos_apical_negg": [0.03402389958500862, 0.06971719115972519, 0.12852472066879272], "cos_Ac_negg": [0.20482957363128662, 0.2466934621334076, 0.23364372551441193], "r_norm": [0.17688536643981934, 0.17364048957824707, 0.17277860641479492]}, {"step": 4500, "epoch": 9, "train_loss": 0.027883945032954216, "cos_r_negg": [0.19688260555267334, 0.2351025640964508, 0.22025971114635468], "cos_apical_negg": [0.029540207237005234, 0.06613779067993164, 0.12443678081035614], "cos_Ac_negg": [0.19691920280456543, 0.23485177755355835, 0.21967586874961853], "r_norm": [0.1727409064769745, 0.16848939657211304, 0.1681206375360489], "ldr": {"dL_sdil": -0.007039323449134827, "dL_bp": -0.01283390074968338, "ratio": 0.5484944590450016}}, {"step": 4600, "epoch": 9, "train_loss": 0.0004145799030084163, "cos_r_negg": [0.2012549340724945, 0.2405065894126892, 0.2255883812904358], "cos_apical_negg": [0.02580380253493786, 0.06926220655441284, 0.12946876883506775], "cos_Ac_negg": [0.20122212171554565, 0.24027664959430695, 0.22505322098731995], "r_norm": [0.18977829813957214, 0.18521910905838013, 0.1842881292104721]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9718, "test_loss": 0.13522891788482666}, {"step": 4700, "epoch": 10, "train_loss": 0.002352712443098426, "cos_r_negg": [0.1847139596939087, 0.21888308227062225, 0.20647931098937988], "cos_apical_negg": [0.018549814820289612, 0.0614742785692215, 0.11759516596794128], "cos_Ac_negg": [0.1847943663597107, 0.2187812477350235, 0.20605726540088654], "r_norm": [0.14868712425231934, 0.1454245150089264, 0.14488908648490906]}, {"step": 4800, "epoch": 10, "train_loss": 0.0063841980881989, "cos_r_negg": [0.1894565224647522, 0.222882479429245, 0.21073979139328003], "cos_apical_negg": [0.02503601834177971, 0.06502237915992737, 0.12016555666923523], "cos_Ac_negg": [0.1895405650138855, 0.22279232740402222, 0.21032766997814178], "r_norm": [0.16727523505687714, 0.16356618702411652, 0.16298474371433258]}, {"step": 4900, "epoch": 10, "train_loss": 0.0010623340494930744, "cos_r_negg": [0.18027281761169434, 0.21277551352977753, 0.20070210099220276], "cos_apical_negg": [0.021039476618170738, 0.06321974098682404, 0.11654315888881683], "cos_Ac_negg": [0.18037821352481842, 0.21267223358154297, 0.20031720399856567], "r_norm": [0.1639031171798706, 0.15994615852832794, 0.1595858931541443]}, {"step": 5000, "epoch": 10, "train_loss": 0.00036228657700121403, "cos_r_negg": [0.18431438505649567, 0.21687689423561096, 0.20406456291675568], "cos_apical_negg": [0.024427860975265503, 0.06379403173923492, 0.1138601005077362], "cos_Ac_negg": [0.18437957763671875, 0.21679972112178802, 0.20367948710918427], "r_norm": [0.17233991622924805, 0.16849377751350403, 0.1679506003856659], "ldr": {"dL_sdil": -0.007889904081821442, "dL_bp": -0.01465548574924469, "ratio": 0.5383584151912583}}, {"step": 5100, "epoch": 10, "train_loss": 0.004313663579523563, "cos_r_negg": [0.1769174039363861, 0.21067111194133759, 0.19968238472938538], "cos_apical_negg": [0.020369667559862137, 0.06167798861861229, 0.11909909546375275], "cos_Ac_negg": [0.1769578456878662, 0.21049445867538452, 0.19923339784145355], "r_norm": [0.1595233976840973, 0.1564694046974182, 0.15605463087558746]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9725, "test_loss": 0.14078899154663085}, {"step": 5200, "epoch": 11, "train_loss": 0.0012677351478487253, "cos_r_negg": [0.16810822486877441, 0.19922301173210144, 0.18810823559761047], "cos_apical_negg": [0.023612264543771744, 0.06200932711362839, 0.11799322813749313], "cos_Ac_negg": [0.16817978024482727, 0.1990954726934433, 0.18768227100372314], "r_norm": [0.16747885942459106, 0.16378852725028992, 0.16306960582733154]}, {"step": 5300, "epoch": 11, "train_loss": 0.002009693067520857, "cos_r_negg": [0.1719294935464859, 0.20394322276115417, 0.1956622302532196], "cos_apical_negg": [0.023588884621858597, 0.06294385343790054, 0.11028198897838593], "cos_Ac_negg": [0.17197218537330627, 0.20385082066059113, 0.19527961313724518], "r_norm": [0.16482827067375183, 0.16162650287151337, 0.16074684262275696]}, {"step": 5400, "epoch": 11, "train_loss": 0.0034146991092711687, "cos_r_negg": [0.16919755935668945, 0.19766053557395935, 0.18918699026107788], "cos_apical_negg": [0.022576574236154556, 0.05874956399202347, 0.10831654816865921], "cos_Ac_negg": [0.16924500465393066, 0.19753403961658478, 0.18878859281539917], "r_norm": [0.16575463116168976, 0.16234257817268372, 0.16134923696517944]}, {"step": 5500, "epoch": 11, "train_loss": 0.0017908771988004446, "cos_r_negg": [0.16837087273597717, 0.19721250236034393, 0.187839075922966], "cos_apical_negg": [0.026299742981791496, 0.05963471531867981, 0.11012455821037292], "cos_Ac_negg": [0.16840282082557678, 0.19710946083068848, 0.18743689358234406], "r_norm": [0.1873452216386795, 0.1830121874809265, 0.18206292390823364], "ldr": {"dL_sdil": -0.009683892130851746, "dL_bp": -0.018616408109664917, "ratio": 0.5201804813155253}}, {"step": 5600, "epoch": 11, "train_loss": 0.0013754954561591148, "cos_r_negg": [0.1636376976966858, 0.19207163155078888, 0.18392041325569153], "cos_apical_negg": [0.02321229875087738, 0.057002753019332886, 0.10404518246650696], "cos_Ac_negg": [0.1637110710144043, 0.19200128316879272, 0.18356746435165405], "r_norm": [0.1753554493188858, 0.1717456877231598, 0.17066563665866852]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9729, "test_loss": 0.14531429138183594}, {"step": 5700, "epoch": 12, "train_loss": 0.0009287337306886911, "cos_r_negg": [0.16079433262348175, 0.18868830800056458, 0.18174007534980774], "cos_apical_negg": [0.02139887399971485, 0.057885367423295975, 0.10542373359203339], "cos_Ac_negg": [0.16087688505649567, 0.18858040869235992, 0.18139316141605377], "r_norm": [0.16926921904087067, 0.16593024134635925, 0.16500943899154663]}, {"step": 5800, "epoch": 12, "train_loss": 0.00030310082365758717, "cos_r_negg": [0.15898418426513672, 0.1867973506450653, 0.1785888373851776], "cos_apical_negg": [0.02138589508831501, 0.05659627914428711, 0.10516968369483948], "cos_Ac_negg": [0.15908759832382202, 0.18671759963035583, 0.1782454252243042], "r_norm": [0.17852672934532166, 0.17494000494480133, 0.1738014966249466]}, {"step": 5900, "epoch": 12, "train_loss": 0.001459339284338057, "cos_r_negg": [0.1598544418811798, 0.1872079074382782, 0.17989645898342133], "cos_apical_negg": [0.023842452093958855, 0.05727141723036766, 0.10314548015594482], "cos_Ac_negg": [0.15993618965148926, 0.18710944056510925, 0.17956651747226715], "r_norm": [0.1763380616903305, 0.17273308336734772, 0.17151713371276855]}, {"step": 6000, "epoch": 12, "train_loss": 0.00025371171068400145, "cos_r_negg": [0.1550983190536499, 0.1812928020954132, 0.17431369423866272], "cos_apical_negg": [0.02227218635380268, 0.05576448142528534, 0.10111168771982193], "cos_Ac_negg": [0.1551620364189148, 0.18118974566459656, 0.17398272454738617], "r_norm": [0.1751830279827118, 0.17173755168914795, 0.17048761248588562], "ldr": {"dL_sdil": -0.008753687143325806, "dL_bp": -0.01805989444255829, "ratio": 0.4847031177932951}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9725, "test_loss": 0.14830777206420898}, {"step": 6100, "epoch": 13, "train_loss": 0.0012382576242089272, "cos_r_negg": [0.1526487171649933, 0.1792762279510498, 0.17127978801727295], "cos_apical_negg": [0.019318794831633568, 0.055017780512571335, 0.09939050674438477], "cos_Ac_negg": [0.15276604890823364, 0.17921492457389832, 0.1709662675857544], "r_norm": [0.17004649341106415, 0.1664789319038391, 0.16549012064933777]}, {"step": 6200, "epoch": 13, "train_loss": 0.00042536837281659245, "cos_r_negg": [0.15009625256061554, 0.17631268501281738, 0.16910406947135925], "cos_apical_negg": [0.02108912542462349, 0.05487648397684097, 0.09703240543603897], "cos_Ac_negg": [0.150213360786438, 0.17624229192733765, 0.16879916191101074], "r_norm": [0.1764199584722519, 0.1728973090648651, 0.1716037541627884]}, {"step": 6300, "epoch": 13, "train_loss": 0.001201061299070716, "cos_r_negg": [0.1477968692779541, 0.1737002432346344, 0.16652484238147736], "cos_apical_negg": [0.020455854013562202, 0.054728034883737564, 0.09616195410490036], "cos_Ac_negg": [0.14792168140411377, 0.1736440360546112, 0.1662227213382721], "r_norm": [0.172653928399086, 0.1693204641342163, 0.16836033761501312]}, {"step": 6400, "epoch": 13, "train_loss": 0.00039902038406580687, "cos_r_negg": [0.1479041576385498, 0.17350591719150543, 0.1664724051952362], "cos_apical_negg": [0.018904881551861763, 0.054554253816604614, 0.09641586244106293], "cos_Ac_negg": [0.14803674817085266, 0.17345182597637177, 0.1661784052848816], "r_norm": [0.17616300284862518, 0.17270249128341675, 0.17161139845848083]}, {"step": 6500, "epoch": 13, "train_loss": 0.00016758985293563455, "cos_r_negg": [0.14824876189231873, 0.1737014353275299, 0.16627953946590424], "cos_apical_negg": [0.018250765278935432, 0.05398929864168167, 0.09398175776004791], "cos_Ac_negg": [0.14837145805358887, 0.17363858222961426, 0.16598375141620636], "r_norm": [0.17000073194503784, 0.1666126400232315, 0.16557259857654572], "ldr": {"dL_sdil": -0.00859159231185913, "dL_bp": -0.018522031605243683, "ratio": 0.4638579878800556}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.973, "test_loss": 0.15130350227355957}, {"step": 6600, "epoch": 14, "train_loss": 0.00037631491431966424, "cos_r_negg": [0.14674055576324463, 0.17132018506526947, 0.1644669622182846], "cos_apical_negg": [0.01886620745062828, 0.05378362163901329, 0.0943530723452568], "cos_Ac_negg": [0.14686821401119232, 0.17125654220581055, 0.1641768217086792], "r_norm": [0.17951610684394836, 0.17604151368141174, 0.17475900053977966]}, {"step": 6700, "epoch": 14, "train_loss": 0.00028938663308508694, "cos_r_negg": [0.14904211461544037, 0.17376399040222168, 0.1669040471315384], "cos_apical_negg": [0.020278243348002434, 0.054608870297670364, 0.09475350379943848], "cos_Ac_negg": [0.14918139576911926, 0.17369869351387024, 0.1666075885295868], "r_norm": [0.1792440116405487, 0.17569801211357117, 0.17455503344535828]}, {"step": 6800, "epoch": 14, "train_loss": 0.0005413379985839128, "cos_r_negg": [0.1450190544128418, 0.16897258162498474, 0.1624162793159485], "cos_apical_negg": [0.019038885831832886, 0.05303763598203659, 0.09342364966869354], "cos_Ac_negg": [0.1451721489429474, 0.16892066597938538, 0.1621500551700592], "r_norm": [0.17567753791809082, 0.17228949069976807, 0.1711525321006775]}, {"step": 6900, "epoch": 14, "train_loss": 0.0007388935191556811, "cos_r_negg": [0.142916738986969, 0.16697654128074646, 0.1604384183883667], "cos_apical_negg": [0.019763367250561714, 0.052394505590200424, 0.09278304874897003], "cos_Ac_negg": [0.14305640757083893, 0.1669260412454605, 0.16018031537532806], "r_norm": [0.1767953783273697, 0.1734650731086731, 0.1722719371318817]}, {"step": 7000, "epoch": 14, "train_loss": 0.000649950176011771, "cos_r_negg": [0.1411450058221817, 0.16502931714057922, 0.15865254402160645], "cos_apical_negg": [0.01981346309185028, 0.053107645362615585, 0.09194252640008926], "cos_Ac_negg": [0.14126035571098328, 0.16497738659381866, 0.15838924050331116], "r_norm": [0.1810358315706253, 0.17778807878494263, 0.17635104060173035], "ldr": {"dL_sdil": -0.01009424775838852, "dL_bp": -0.02193962037563324, "ratio": 0.460092179607605}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9727, "test_loss": 0.15496361064910888}], "final": {"test_acc": 0.9727, "test_loss": 0.15496361064910888, "wall_s": 59.02428102493286, "cos_r_negg": [0.14169630408287048, 0.1655777096748352, 0.1592366099357605], "cos_apical_negg": [0.018531912937760353, 0.052661314606666565, 0.09144338220357895], "cos_Ac_negg": [0.14182420074939728, 0.1655307412147522, 0.15897312760353088]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p05_res0_s0.json b/results/diag_nuis_low_rho0p05_res0_s0.json new file mode 100644 index 0000000..7b3dd22 --- /dev/null +++ b/results/diag_nuis_low_rho0p05_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.05, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p05_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.018859541043639183, -0.025855394080281258, 0.03358403220772743], "cos_apical_negg": [-0.018884912133216858, -0.02600540593266487, 0.032936543226242065], "cos_Ac_negg": [-0.018859509378671646, -0.025855395942926407, 0.03358401358127594], "r_norm": [4.89775276184082, 4.8266096115112305, 4.823806285858154], "ldr": {"dL_sdil": -0.008686065673828125, "dL_bp": -0.23511672019958496, "ratio": 0.03694363236461759}}, {"step": 100, "epoch": 0, "train_loss": 0.23041948676109314, "cos_r_negg": [0.4495658278465271, 0.438557505607605, 0.4032081663608551], "cos_apical_negg": [0.23425103724002838, 0.2305426448583603, 0.20068371295928955], "cos_Ac_negg": [0.4495430588722229, 0.43857237696647644, 0.4032664895057678], "r_norm": [0.8637661337852478, 0.8459800481796265, 0.8417215943336487]}, {"step": 200, "epoch": 0, "train_loss": 0.31900668144226074, "cos_r_negg": [0.4667399525642395, 0.4753493368625641, 0.42676639556884766], "cos_apical_negg": [0.20321866869926453, 0.253325879573822, 0.2679175138473511], "cos_Ac_negg": [0.4667106568813324, 0.4753454029560089, 0.4267539381980896], "r_norm": [0.7176737785339355, 0.7017897367477417, 0.6974855065345764]}, {"step": 300, "epoch": 0, "train_loss": 0.22656193375587463, "cos_r_negg": [0.48032641410827637, 0.4936434328556061, 0.45862603187561035], "cos_apical_negg": [0.2153615951538086, 0.24147266149520874, 0.22866767644882202], "cos_Ac_negg": [0.4803088307380676, 0.4936337172985077, 0.45863789319992065], "r_norm": [0.632865309715271, 0.6213756799697876, 0.6162056922912598]}, {"step": 400, "epoch": 0, "train_loss": 0.14908280968666077, "cos_r_negg": [0.4908028542995453, 0.47934049367904663, 0.47111719846725464], "cos_apical_negg": [0.17224785685539246, 0.2163817584514618, 0.2442982941865921], "cos_Ac_negg": [0.4908183813095093, 0.4794113337993622, 0.4711703956127167], "r_norm": [0.5587549209594727, 0.54682457447052, 0.5428429841995239]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9384, "test_loss": 0.22432612457275392}, {"step": 500, "epoch": 1, "train_loss": 0.1585507094860077, "cos_r_negg": [0.499206006526947, 0.4916081726551056, 0.4651931822299957], "cos_apical_negg": [0.14503195881843567, 0.19192549586296082, 0.20011019706726074], "cos_Ac_negg": [0.49918317794799805, 0.491603285074234, 0.46515461802482605], "r_norm": [0.4864939749240875, 0.47676655650138855, 0.47347795963287354], "ldr": {"dL_sdil": -0.007810428738594055, "dL_bp": -0.006880521774291992, "ratio": 1.1351506462455387}}, {"step": 600, "epoch": 1, "train_loss": 0.10101154446601868, "cos_r_negg": [0.45263826847076416, 0.4400579631328583, 0.43320292234420776], "cos_apical_negg": [0.16803187131881714, 0.20452865958213806, 0.25366052985191345], "cos_Ac_negg": [0.452607125043869, 0.44006866216659546, 0.4331720173358917], "r_norm": [0.5107088685035706, 0.5015692710876465, 0.4965692460536957]}, {"step": 700, "epoch": 1, "train_loss": 0.18444254994392395, "cos_r_negg": [0.4470541477203369, 0.4408856928348541, 0.3460756242275238], "cos_apical_negg": [0.17038634419441223, 0.1927310973405838, 0.21171502768993378], "cos_Ac_negg": [0.44705718755722046, 0.440972238779068, 0.3459165096282959], "r_norm": [0.519079864025116, 0.50715571641922, 0.5048587322235107]}, {"step": 800, "epoch": 1, "train_loss": 0.13507409393787384, "cos_r_negg": [0.44011786580085754, 0.45421677827835083, 0.49226370453834534], "cos_apical_negg": [0.13460895419120789, 0.16802546381950378, 0.2506113648414612], "cos_Ac_negg": [0.44009929895401, 0.4542716145515442, 0.4920908510684967], "r_norm": [0.40946710109710693, 0.40172988176345825, 0.39880478382110596]}, {"step": 900, "epoch": 1, "train_loss": 0.19447556138038635, "cos_r_negg": [0.48749077320098877, 0.49739038944244385, 0.5107520818710327], "cos_apical_negg": [0.16051051020622253, 0.19374898076057434, 0.24381804466247559], "cos_Ac_negg": [0.48747438192367554, 0.49736422300338745, 0.5107160806655884], "r_norm": [0.45803895592689514, 0.44658830761909485, 0.4441685080528259]}, {"epoch_end": 1, "step": 938, "test_acc": 0.946, "test_loss": 0.19735396728515625}, {"step": 1000, "epoch": 2, "train_loss": 0.0457681342959404, "cos_r_negg": [0.4296809434890747, 0.4640665650367737, 0.4848756790161133], "cos_apical_negg": [0.15173259377479553, 0.14346879720687866, 0.2658452093601227], "cos_Ac_negg": [0.42959991097450256, 0.46401816606521606, 0.48476260900497437], "r_norm": [0.4119800627231598, 0.40227270126342773, 0.39989402890205383], "ldr": {"dL_sdil": -0.006961911916732788, "dL_bp": -0.006557494401931763, "ratio": 1.0616725672967238}}, {"step": 1100, "epoch": 2, "train_loss": 0.21052515506744385, "cos_r_negg": [0.4520382583141327, 0.5036782622337341, 0.47959625720977783], "cos_apical_negg": [0.12356582283973694, 0.22508955001831055, 0.2550658583641052], "cos_Ac_negg": [0.4519824981689453, 0.5036282539367676, 0.4795272946357727], "r_norm": [0.407164990901947, 0.3967611491680145, 0.39590781927108765]}, {"step": 1200, "epoch": 2, "train_loss": 0.1942715048789978, "cos_r_negg": [0.4031233489513397, 0.4293915033340454, 0.47051721811294556], "cos_apical_negg": [0.11561797559261322, 0.1766512393951416, 0.29191339015960693], "cos_Ac_negg": [0.40306222438812256, 0.429315984249115, 0.47041720151901245], "r_norm": [0.426477313041687, 0.4153887629508972, 0.41381675004959106]}, {"step": 1300, "epoch": 2, "train_loss": 0.15486928820610046, "cos_r_negg": [0.4573405981063843, 0.5050272345542908, 0.5007753372192383], "cos_apical_negg": [0.12150683254003525, 0.17284974455833435, 0.2864055633544922], "cos_Ac_negg": [0.45729613304138184, 0.505011796951294, 0.5007431507110596], "r_norm": [0.40167248249053955, 0.39097899198532104, 0.3881244957447052]}, {"step": 1400, "epoch": 2, "train_loss": 0.17530547082424164, "cos_r_negg": [0.43483632802963257, 0.4593575894832611, 0.45625531673431396], "cos_apical_negg": [0.1269143521785736, 0.14315594732761383, 0.28137022256851196], "cos_Ac_negg": [0.43475961685180664, 0.45924699306488037, 0.4560891389846802], "r_norm": [0.36618512868881226, 0.35830798745155334, 0.3549748659133911]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9586, "test_loss": 0.15918482398986816}, {"step": 1500, "epoch": 3, "train_loss": 0.10709927976131439, "cos_r_negg": [0.39128565788269043, 0.43114301562309265, 0.5231647491455078], "cos_apical_negg": [0.13121604919433594, 0.17816883325576782, 0.2994878888130188], "cos_Ac_negg": [0.3912202715873718, 0.4310793876647949, 0.5230669975280762], "r_norm": [0.3276882469654083, 0.3205508589744568, 0.31767234206199646], "ldr": {"dL_sdil": -0.005378603935241699, "dL_bp": -0.005034893751144409, "ratio": 1.0682656280520648}}, {"step": 1600, "epoch": 3, "train_loss": 0.09595062583684921, "cos_r_negg": [0.4049469828605652, 0.452851802110672, 0.5016907453536987], "cos_apical_negg": [0.09041190147399902, 0.17017976939678192, 0.24415323138237], "cos_Ac_negg": [0.40489840507507324, 0.45268815755844116, 0.5016027092933655], "r_norm": [0.30148738622665405, 0.29396745562553406, 0.29216906428337097]}, {"step": 1700, "epoch": 3, "train_loss": 0.08881118893623352, "cos_r_negg": [0.42339393496513367, 0.471749871969223, 0.5213449001312256], "cos_apical_negg": [0.11011786758899689, 0.1666950285434723, 0.3406198024749756], "cos_Ac_negg": [0.4233313202857971, 0.4716889262199402, 0.5212584733963013], "r_norm": [0.3786996603012085, 0.3675631880760193, 0.36543840169906616]}, {"step": 1800, "epoch": 3, "train_loss": 0.048381127417087555, "cos_r_negg": [0.39093923568725586, 0.4543036222457886, 0.4938172996044159], "cos_apical_negg": [0.10372577607631683, 0.17405515909194946, 0.2991994023323059], "cos_Ac_negg": [0.39087870717048645, 0.4542158246040344, 0.493794322013855], "r_norm": [0.3531760573387146, 0.34701007604599, 0.3419577479362488]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9548, "test_loss": 0.17337781944274902}, {"step": 1900, "epoch": 4, "train_loss": 0.05800715088844299, "cos_r_negg": [0.4079931676387787, 0.452181339263916, 0.5453986525535583], "cos_apical_negg": [0.10339056700468063, 0.18660768866539001, 0.3291071057319641], "cos_Ac_negg": [0.40799185633659363, 0.4520745873451233, 0.5452721118927002], "r_norm": [0.33118006587028503, 0.32496699690818787, 0.3217678368091583]}, {"step": 2000, "epoch": 4, "train_loss": 0.17575505375862122, "cos_r_negg": [0.37762051820755005, 0.428901731967926, 0.4976288974285126], "cos_apical_negg": [0.0639958530664444, 0.14581869542598724, 0.26892420649528503], "cos_Ac_negg": [0.37761449813842773, 0.42886531352996826, 0.4976373612880707], "r_norm": [0.28962695598602295, 0.2811424732208252, 0.2803577780723572], "ldr": {"dL_sdil": -0.004555374383926392, "dL_bp": -0.004821710288524628, "ratio": 0.9447631880264356}}, {"step": 2100, "epoch": 4, "train_loss": 0.05293913930654526, "cos_r_negg": [0.402860552072525, 0.4590412676334381, 0.5967625379562378], "cos_apical_negg": [0.10692726820707321, 0.19157880544662476, 0.39980432391166687], "cos_Ac_negg": [0.4028604030609131, 0.45902958512306213, 0.5967142581939697], "r_norm": [0.3243647515773773, 0.3144170939922333, 0.3137452304363251]}, {"step": 2200, "epoch": 4, "train_loss": 0.011271443217992783, "cos_r_negg": [0.3920633792877197, 0.46538612246513367, 0.506847620010376], "cos_apical_negg": [0.07115926593542099, 0.1509314626455307, 0.34705302119255066], "cos_Ac_negg": [0.392006516456604, 0.46531832218170166, 0.506816029548645], "r_norm": [0.28227996826171875, 0.2757565975189209, 0.2731337547302246]}, {"step": 2300, "epoch": 4, "train_loss": 0.2430552840232849, "cos_r_negg": [0.3598286509513855, 0.4378488063812256, 0.47929275035858154], "cos_apical_negg": [0.09724930673837662, 0.16499607264995575, 0.30086958408355713], "cos_Ac_negg": [0.35976845026016235, 0.437702476978302, 0.4792115390300751], "r_norm": [0.28910478949546814, 0.28181856870651245, 0.2799420952796936]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9605, "test_loss": 0.16457430114746094}, {"step": 2400, "epoch": 5, "train_loss": 0.10465245693922043, "cos_r_negg": [0.34559330344200134, 0.40749529004096985, 0.48866987228393555], "cos_apical_negg": [0.09975217282772064, 0.13326816260814667, 0.2792685925960541], "cos_Ac_negg": [0.34555360674858093, 0.40741896629333496, 0.48864057660102844], "r_norm": [0.2857345938682556, 0.27781054377555847, 0.27678734064102173]}, {"step": 2500, "epoch": 5, "train_loss": 0.04419015720486641, "cos_r_negg": [0.3481239378452301, 0.40166759490966797, 0.4526602029800415], "cos_apical_negg": [0.08520470559597015, 0.1418285220861435, 0.2726573944091797], "cos_Ac_negg": [0.348020076751709, 0.40157097578048706, 0.4526427388191223], "r_norm": [0.31252631545066833, 0.3041263222694397, 0.30180683732032776], "ldr": {"dL_sdil": -0.010241255164146423, "dL_bp": -0.010050714015960693, "ratio": 1.0189579713324992}}, {"step": 2600, "epoch": 5, "train_loss": 0.08398325741291046, "cos_r_negg": [0.32912468910217285, 0.3919678330421448, 0.46308478713035583], "cos_apical_negg": [0.0678214430809021, 0.12996843457221985, 0.32816237211227417], "cos_Ac_negg": [0.3290342688560486, 0.3918955326080322, 0.4629836082458496], "r_norm": [0.30338242650032043, 0.2963986396789551, 0.29207080602645874]}, {"step": 2700, "epoch": 5, "train_loss": 0.06364405900239944, "cos_r_negg": [0.367350697517395, 0.42601579427719116, 0.4927910566329956], "cos_apical_negg": [0.08950985968112946, 0.15796583890914917, 0.3116663992404938], "cos_Ac_negg": [0.3673328757286072, 0.4259517788887024, 0.49272680282592773], "r_norm": [0.2901665270328522, 0.28244537115097046, 0.28005003929138184]}, {"step": 2800, "epoch": 5, "train_loss": 0.09003597497940063, "cos_r_negg": [0.3629224896430969, 0.4101867973804474, 0.48388198018074036], "cos_apical_negg": [0.1068108081817627, 0.1504707932472229, 0.33690547943115234], "cos_Ac_negg": [0.362865149974823, 0.41013211011886597, 0.48381614685058594], "r_norm": [0.2653420567512512, 0.2571680247783661, 0.25444555282592773]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.96, "test_loss": 0.16635609436035156}, {"step": 2900, "epoch": 6, "train_loss": 0.01722564920783043, "cos_r_negg": [0.3429734706878662, 0.39396265149116516, 0.46862703561782837], "cos_apical_negg": [0.06533334404230118, 0.1568993330001831, 0.30519652366638184], "cos_Ac_negg": [0.3429383635520935, 0.39387381076812744, 0.4686115086078644], "r_norm": [0.2505260109901428, 0.2434178739786148, 0.24215924739837646]}, {"step": 3000, "epoch": 6, "train_loss": 0.254302442073822, "cos_r_negg": [0.32735443115234375, 0.37727364897727966, 0.4561707675457001], "cos_apical_negg": [0.08991317451000214, 0.14201776683330536, 0.2580964267253876], "cos_Ac_negg": [0.32726866006851196, 0.3771824240684509, 0.45612066984176636], "r_norm": [0.24009299278259277, 0.2335033118724823, 0.23213297128677368], "ldr": {"dL_sdil": -0.004550516605377197, "dL_bp": -0.0057260096073150635, "ratio": 0.7947099144863193}}, {"step": 3100, "epoch": 6, "train_loss": 0.0999719500541687, "cos_r_negg": [0.3422325551509857, 0.40003567934036255, 0.49887576699256897], "cos_apical_negg": [0.09119797497987747, 0.18068227171897888, 0.29104316234588623], "cos_Ac_negg": [0.34217894077301025, 0.3999730348587036, 0.49884071946144104], "r_norm": [0.26564186811447144, 0.2583104968070984, 0.2559945583343506]}, {"step": 3200, "epoch": 6, "train_loss": 0.10090172290802002, "cos_r_negg": [0.36575227975845337, 0.4082220196723938, 0.4817204177379608], "cos_apical_negg": [0.05126918852329254, 0.1369341015815735, 0.30862876772880554], "cos_Ac_negg": [0.36572355031967163, 0.408191442489624, 0.48167240619659424], "r_norm": [0.2422788292169571, 0.23367345333099365, 0.2331787645816803]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9609, "test_loss": 0.15475603256225587}, {"step": 3300, "epoch": 7, "train_loss": 0.04708199203014374, "cos_r_negg": [0.3485734462738037, 0.4070676267147064, 0.43859055638313293], "cos_apical_negg": [0.09018873423337936, 0.145110622048378, 0.2820039689540863], "cos_Ac_negg": [0.3484886884689331, 0.40698564052581787, 0.4385114014148712], "r_norm": [0.24502712488174438, 0.23884201049804688, 0.23797070980072021]}, {"step": 3400, "epoch": 7, "train_loss": 0.0928717777132988, "cos_r_negg": [0.327043354511261, 0.3553234040737152, 0.4023120403289795], "cos_apical_negg": [0.06541968137025833, 0.15065276622772217, 0.25840258598327637], "cos_Ac_negg": [0.3269844651222229, 0.3552144765853882, 0.40223097801208496], "r_norm": [0.2611147463321686, 0.2536992132663727, 0.25139155983924866]}, {"step": 3500, "epoch": 7, "train_loss": 0.039609394967556, "cos_r_negg": [0.31533244252204895, 0.3724842667579651, 0.4049232602119446], "cos_apical_negg": [0.06721514463424683, 0.14995111525058746, 0.2615854740142822], "cos_Ac_negg": [0.31531035900115967, 0.37240034341812134, 0.4049084186553955], "r_norm": [0.2457149624824524, 0.23849642276763916, 0.2367725670337677], "ldr": {"dL_sdil": -0.004415854811668396, "dL_bp": -0.006016790866851807, "ratio": 0.7339219376888405}}, {"step": 3600, "epoch": 7, "train_loss": 0.02590642124414444, "cos_r_negg": [0.3316219747066498, 0.38672149181365967, 0.444937527179718], "cos_apical_negg": [0.06654739379882812, 0.1582154780626297, 0.3144466280937195], "cos_Ac_negg": [0.3315872550010681, 0.3866783380508423, 0.4448981285095215], "r_norm": [0.25783854722976685, 0.24984697997570038, 0.24949876964092255]}, {"step": 3700, "epoch": 7, "train_loss": 0.04980098456144333, "cos_r_negg": [0.3282231092453003, 0.3729826807975769, 0.4369191527366638], "cos_apical_negg": [0.07096027582883835, 0.12717995047569275, 0.28872522711753845], "cos_Ac_negg": [0.3281758427619934, 0.37288156151771545, 0.4368581175804138], "r_norm": [0.24191997945308685, 0.23422472178936005, 0.2330273985862732]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9616, "test_loss": 0.16512727928161622}, {"step": 3800, "epoch": 8, "train_loss": 0.042860131710767746, "cos_r_negg": [0.302994042634964, 0.35327497124671936, 0.3922566771507263], "cos_apical_negg": [0.08146801590919495, 0.12556783854961395, 0.24825069308280945], "cos_Ac_negg": [0.3029629588127136, 0.35318559408187866, 0.39219945669174194], "r_norm": [0.24947410821914673, 0.24094158411026, 0.24069556593894958]}, {"step": 3900, "epoch": 8, "train_loss": 0.19851869344711304, "cos_r_negg": [0.30765265226364136, 0.3734223544597626, 0.4112052917480469], "cos_apical_negg": [0.0694645345211029, 0.10811938345432281, 0.2396838665008545], "cos_Ac_negg": [0.3075825572013855, 0.37338367104530334, 0.4111306071281433], "r_norm": [0.2629826068878174, 0.2532013952732086, 0.2535835802555084]}, {"step": 4000, "epoch": 8, "train_loss": 0.0993446558713913, "cos_r_negg": [0.30676549673080444, 0.36268967390060425, 0.40528762340545654], "cos_apical_negg": [0.06921800971031189, 0.14448705315589905, 0.23392590880393982], "cos_Ac_negg": [0.3066401481628418, 0.3625635504722595, 0.4051450788974762], "r_norm": [0.250262975692749, 0.24086323380470276, 0.24081017076969147], "ldr": {"dL_sdil": -0.005112171173095703, "dL_bp": -0.007170885801315308, "ratio": 0.7129065104004322}}, {"step": 4100, "epoch": 8, "train_loss": 0.028416089713573456, "cos_r_negg": [0.2816343307495117, 0.33587607741355896, 0.402446448802948], "cos_apical_negg": [0.07643571496009827, 0.1203567162156105, 0.20532774925231934], "cos_Ac_negg": [0.28160321712493896, 0.3357827961444855, 0.4024065136909485], "r_norm": [0.23181551694869995, 0.2243141531944275, 0.22228580713272095]}, {"step": 4200, "epoch": 8, "train_loss": 0.11943857371807098, "cos_r_negg": [0.2794262170791626, 0.33612802624702454, 0.34654927253723145], "cos_apical_negg": [0.06141865253448486, 0.12693995237350464, 0.26508644223213196], "cos_Ac_negg": [0.2793498635292053, 0.3360079228878021, 0.3463972806930542], "r_norm": [0.23204664885997772, 0.22428905963897705, 0.22379128634929657]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9624, "test_loss": 0.1759301212310791}, {"step": 4300, "epoch": 9, "train_loss": 0.03324798494577408, "cos_r_negg": [0.2684468924999237, 0.34543082118034363, 0.4006440043449402], "cos_apical_negg": [0.0746016576886177, 0.11710374057292938, 0.2527583837509155], "cos_Ac_negg": [0.26840752363204956, 0.3453753590583801, 0.40066009759902954], "r_norm": [0.22844037413597107, 0.22068721055984497, 0.21950975060462952]}, {"step": 4400, "epoch": 9, "train_loss": 0.06485579907894135, "cos_r_negg": [0.2945616841316223, 0.3529946208000183, 0.3829984664916992], "cos_apical_negg": [0.0716085284948349, 0.1288042962551117, 0.2324242889881134], "cos_Ac_negg": [0.2945319712162018, 0.35290783643722534, 0.3829905390739441], "r_norm": [0.24532450735569, 0.2356562614440918, 0.23506540060043335]}, {"step": 4500, "epoch": 9, "train_loss": 0.16204960644245148, "cos_r_negg": [0.288923978805542, 0.3374084234237671, 0.38374823331832886], "cos_apical_negg": [0.06583872437477112, 0.1379699856042862, 0.23359335958957672], "cos_Ac_negg": [0.2888481616973877, 0.3372815251350403, 0.3836948275566101], "r_norm": [0.25064408779144287, 0.24283459782600403, 0.2415226697921753], "ldr": {"dL_sdil": -0.005103409290313721, "dL_bp": -0.008329182863235474, "ratio": 0.6127142811139219}}, {"step": 4600, "epoch": 9, "train_loss": 0.019619045779109, "cos_r_negg": [0.27797144651412964, 0.32512104511260986, 0.3862626552581787], "cos_apical_negg": [0.062265969812870026, 0.12815988063812256, 0.217443585395813], "cos_Ac_negg": [0.277919739484787, 0.3250352144241333, 0.38623300194740295], "r_norm": [0.2724788784980774, 0.2608310282230377, 0.2602118253707886]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9627, "test_loss": 0.17319459648132324}, {"step": 4700, "epoch": 10, "train_loss": 0.06530911475419998, "cos_r_negg": [0.2737928628921509, 0.3167988657951355, 0.3624982535839081], "cos_apical_negg": [0.05613955855369568, 0.10609345883131027, 0.22260531783103943], "cos_Ac_negg": [0.2737599015235901, 0.31674450635910034, 0.3624531030654907], "r_norm": [0.23430897295475006, 0.22494898736476898, 0.2250165343284607]}, {"step": 4800, "epoch": 10, "train_loss": 0.03400012105703354, "cos_r_negg": [0.26803454756736755, 0.3138619661331177, 0.3509948253631592], "cos_apical_negg": [0.07172602415084839, 0.10853955149650574, 0.21228213608264923], "cos_Ac_negg": [0.2680264413356781, 0.3138158619403839, 0.3509902060031891], "r_norm": [0.22175981104373932, 0.21401754021644592, 0.21272143721580505]}, {"step": 4900, "epoch": 10, "train_loss": 0.041966889053583145, "cos_r_negg": [0.28054535388946533, 0.3304911255836487, 0.3454379141330719], "cos_apical_negg": [0.08751124143600464, 0.12402290850877762, 0.20934724807739258], "cos_Ac_negg": [0.28044596314430237, 0.33033812046051025, 0.34538978338241577], "r_norm": [0.2642386257648468, 0.25565317273139954, 0.2547498941421509]}, {"step": 5000, "epoch": 10, "train_loss": 0.016221091151237488, "cos_r_negg": [0.24009081721305847, 0.28166189789772034, 0.33274024724960327], "cos_apical_negg": [0.05780601501464844, 0.10293823480606079, 0.1679936647415161], "cos_Ac_negg": [0.23999138176441193, 0.28161919116973877, 0.3326970934867859], "r_norm": [0.26896214485168457, 0.25706785917282104, 0.25718507170677185], "ldr": {"dL_sdil": -0.007662743330001831, "dL_bp": -0.011037737131118774, "ratio": 0.6942313663548122}}, {"step": 5100, "epoch": 10, "train_loss": 0.088715098798275, "cos_r_negg": [0.26628926396369934, 0.3186708986759186, 0.3478989005088806], "cos_apical_negg": [0.06275370717048645, 0.13171781599521637, 0.17688298225402832], "cos_Ac_negg": [0.26624634861946106, 0.31865519285202026, 0.34788453578948975], "r_norm": [0.222150981426239, 0.21377214789390564, 0.21335174143314362]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9629, "test_loss": 0.17506438331604005}, {"step": 5200, "epoch": 11, "train_loss": 0.02393886260688305, "cos_r_negg": [0.25286173820495605, 0.29233118891716003, 0.34782710671424866], "cos_apical_negg": [0.04996640235185623, 0.0957651287317276, 0.15935391187667847], "cos_Ac_negg": [0.25284072756767273, 0.2922801375389099, 0.34784412384033203], "r_norm": [0.25562959909439087, 0.24406994879245758, 0.24479852616786957]}, {"step": 5300, "epoch": 11, "train_loss": 0.06693704426288605, "cos_r_negg": [0.24851581454277039, 0.2895551025867462, 0.3296932578086853], "cos_apical_negg": [0.05208826810121536, 0.09988594800233841, 0.15827281773090363], "cos_Ac_negg": [0.24855650961399078, 0.2895391583442688, 0.3296216130256653], "r_norm": [0.22721660137176514, 0.21903525292873383, 0.21780428290367126]}, {"step": 5400, "epoch": 11, "train_loss": 0.10611436516046524, "cos_r_negg": [0.23775961995124817, 0.2720891237258911, 0.30760571360588074], "cos_apical_negg": [0.05060301721096039, 0.07948607206344604, 0.1560310274362564], "cos_Ac_negg": [0.23772010207176208, 0.27209165692329407, 0.3075942397117615], "r_norm": [0.20916828513145447, 0.20119571685791016, 0.20080481469631195]}, {"step": 5500, "epoch": 11, "train_loss": 0.06458264589309692, "cos_r_negg": [0.21913184225559235, 0.26514941453933716, 0.3119129240512848], "cos_apical_negg": [0.04851216822862625, 0.08891189843416214, 0.15051916241645813], "cos_Ac_negg": [0.21908321976661682, 0.2651149034500122, 0.31189125776290894], "r_norm": [0.2237250655889511, 0.21489155292510986, 0.2150355577468872], "ldr": {"dL_sdil": -0.005355149507522583, "dL_bp": -0.007999122142791748, "ratio": 0.6694671505107933}}, {"step": 5600, "epoch": 11, "train_loss": 0.07114279270172119, "cos_r_negg": [0.2423516809940338, 0.27774497866630554, 0.3236916959285736], "cos_apical_negg": [0.041089002043008804, 0.0877259150147438, 0.14579185843467712], "cos_Ac_negg": [0.24226784706115723, 0.2776843309402466, 0.32356932759284973], "r_norm": [0.25601786375045776, 0.24498695135116577, 0.24455681443214417]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9654, "test_loss": 0.17612462768554688}, {"step": 5700, "epoch": 12, "train_loss": 0.022405415773391724, "cos_r_negg": [0.23077552020549774, 0.26930665969848633, 0.335257351398468], "cos_apical_negg": [0.053240492939949036, 0.11175261437892914, 0.1889306753873825], "cos_Ac_negg": [0.23069755733013153, 0.26925185322761536, 0.33523252606391907], "r_norm": [0.23330043256282806, 0.22436398267745972, 0.22454747557640076]}, {"step": 5800, "epoch": 12, "train_loss": 0.023805033415555954, "cos_r_negg": [0.22505289316177368, 0.26815885305404663, 0.3219186067581177], "cos_apical_negg": [0.047677524387836456, 0.08665858209133148, 0.20106050372123718], "cos_Ac_negg": [0.2250097393989563, 0.2681529223918915, 0.32175537943840027], "r_norm": [0.2285201996564865, 0.220094233751297, 0.21931485831737518]}, {"step": 5900, "epoch": 12, "train_loss": 0.05272120609879494, "cos_r_negg": [0.21296633780002594, 0.25359073281288147, 0.3151461184024811], "cos_apical_negg": [0.046626731753349304, 0.07314882427453995, 0.15071454644203186], "cos_Ac_negg": [0.21289929747581482, 0.2535213232040405, 0.31506434082984924], "r_norm": [0.23547467589378357, 0.22570133209228516, 0.2249888777732849]}, {"step": 6000, "epoch": 12, "train_loss": 0.03989332914352417, "cos_r_negg": [0.2101084291934967, 0.2545337677001953, 0.3286658823490143], "cos_apical_negg": [0.05492443963885307, 0.0689796507358551, 0.16387903690338135], "cos_Ac_negg": [0.21004396677017212, 0.25446656346321106, 0.3286169469356537], "r_norm": [0.2563685178756714, 0.24540486931800842, 0.24480032920837402], "ldr": {"dL_sdil": -0.008207187056541443, "dL_bp": -0.012436360120773315, "ratio": 0.6599348182940127}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9635, "test_loss": 0.17738923530578612}, {"step": 6100, "epoch": 13, "train_loss": 0.028543120250105858, "cos_r_negg": [0.22327300906181335, 0.26672330498695374, 0.30960410833358765], "cos_apical_negg": [0.039237797260284424, 0.0947365090250969, 0.16351380944252014], "cos_Ac_negg": [0.2232847511768341, 0.26667866110801697, 0.3095558285713196], "r_norm": [0.24001407623291016, 0.23085787892341614, 0.2299347221851349]}, {"step": 6200, "epoch": 13, "train_loss": 0.004910183139145374, "cos_r_negg": [0.20307967066764832, 0.24216187000274658, 0.3206261098384857], "cos_apical_negg": [0.04632951691746712, 0.08182285726070404, 0.13545255362987518], "cos_Ac_negg": [0.2030533254146576, 0.24210405349731445, 0.3205753266811371], "r_norm": [0.2204481065273285, 0.2112005203962326, 0.21163447201251984]}, {"step": 6300, "epoch": 13, "train_loss": 0.03648144751787186, "cos_r_negg": [0.21642978489398956, 0.24986572563648224, 0.31801748275756836], "cos_apical_negg": [0.050873544067144394, 0.08132611215114594, 0.1632038950920105], "cos_Ac_negg": [0.2163843810558319, 0.24972546100616455, 0.31798988580703735], "r_norm": [0.21890299022197723, 0.20938771963119507, 0.20971077680587769]}, {"step": 6400, "epoch": 13, "train_loss": 0.056792933493852615, "cos_r_negg": [0.22756576538085938, 0.2788909077644348, 0.3360409438610077], "cos_apical_negg": [0.05521797388792038, 0.11620445549488068, 0.17335009574890137], "cos_Ac_negg": [0.22754937410354614, 0.27884581685066223, 0.33595243096351624], "r_norm": [0.2516174912452698, 0.23961853981018066, 0.24066844582557678]}, {"step": 6500, "epoch": 13, "train_loss": 0.004034698475152254, "cos_r_negg": [0.19713154435157776, 0.23722998797893524, 0.2868044674396515], "cos_apical_negg": [0.059556350111961365, 0.08835192024707794, 0.14189296960830688], "cos_Ac_negg": [0.19704122841358185, 0.23709474503993988, 0.28666165471076965], "r_norm": [0.22154957056045532, 0.21172425150871277, 0.21188592910766602], "ldr": {"dL_sdil": -0.006559327244758606, "dL_bp": -0.010899052023887634, "ratio": 0.6018254826550437}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9652, "test_loss": 0.17831709709167481}, {"step": 6600, "epoch": 14, "train_loss": 0.007906114682555199, "cos_r_negg": [0.20714542269706726, 0.24214006960391998, 0.3080390989780426], "cos_apical_negg": [0.04364532604813576, 0.07997705787420273, 0.11901251971721649], "cos_Ac_negg": [0.20708507299423218, 0.24202898144721985, 0.30800262093544006], "r_norm": [0.22210271656513214, 0.21261802315711975, 0.21386681497097015]}, {"step": 6700, "epoch": 14, "train_loss": 0.019950084388256073, "cos_r_negg": [0.20827743411064148, 0.24096807837486267, 0.29454314708709717], "cos_apical_negg": [0.04482191056013107, 0.07248908281326294, 0.14117850363254547], "cos_Ac_negg": [0.20824681222438812, 0.24085980653762817, 0.2944616675376892], "r_norm": [0.22775590419769287, 0.2181779444217682, 0.21792297065258026]}, {"step": 6800, "epoch": 14, "train_loss": 0.04818455129861832, "cos_r_negg": [0.20619240403175354, 0.23296505212783813, 0.28828147053718567], "cos_apical_negg": [0.04541243985295296, 0.08153180778026581, 0.1571035534143448], "cos_Ac_negg": [0.20624148845672607, 0.23292222619056702, 0.28811535239219666], "r_norm": [0.22276106476783752, 0.21435806155204773, 0.2135123312473297]}, {"step": 6900, "epoch": 14, "train_loss": 0.021439220756292343, "cos_r_negg": [0.19076603651046753, 0.22330279648303986, 0.273693323135376], "cos_apical_negg": [0.0345194935798645, 0.06461836397647858, 0.1375429481267929], "cos_Ac_negg": [0.19075411558151245, 0.22319884598255157, 0.27354273200035095], "r_norm": [0.2093660980463028, 0.20056773722171783, 0.20048120617866516]}, {"step": 7000, "epoch": 14, "train_loss": 0.03687659278512001, "cos_r_negg": [0.21563002467155457, 0.25248804688453674, 0.28878653049468994], "cos_apical_negg": [0.05158090591430664, 0.08302133530378342, 0.15268759429454803], "cos_Ac_negg": [0.21567964553833008, 0.25236842036247253, 0.288669228553772], "r_norm": [0.24374818801879883, 0.2332955002784729, 0.23362621665000916], "ldr": {"dL_sdil": -0.00705382227897644, "dL_bp": -0.012078821659088135, "ratio": 0.5839826498033546}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9648, "test_loss": 0.18651994438171388}], "final": {"test_acc": 0.9648, "test_loss": 0.18651994438171388, "wall_s": 41.70946145057678, "cos_r_negg": [0.20066574215888977, 0.2392016500234604, 0.30216458439826965], "cos_apical_negg": [0.0591607540845871, 0.08060980588197708, 0.13743393123149872], "cos_Ac_negg": [0.2006959319114685, 0.2390655279159546, 0.3021112382411957]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p05_res1_s0.json b/results/diag_nuis_low_rho0p05_res1_s0.json new file mode 100644 index 0000000..ce6314e --- /dev/null +++ b/results/diag_nuis_low_rho0p05_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.05, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p05_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.019516214728355408, -0.02552604302763939, 0.03376482054591179], "cos_apical_negg": [-0.019511956721544266, -0.025701938197016716, 0.033109668642282486], "cos_Ac_negg": [-0.019516170024871826, -0.02552604116499424, 0.03376481309533119], "r_norm": [4.898174285888672, 4.826803207397461, 4.823856830596924], "ldr": {"dL_sdil": -0.009198665618896484, "dL_bp": -0.2406785488128662, "ratio": 0.038219715318495974}}, {"step": 100, "epoch": 0, "train_loss": 0.20672431588172913, "cos_r_negg": [0.45616453886032104, 0.381319135427475, 0.36034852266311646], "cos_apical_negg": [0.19364020228385925, 0.175805926322937, 0.21464210748672485], "cos_Ac_negg": [0.4564410448074341, 0.38162362575531006, 0.3605561852455139], "r_norm": [0.8243943452835083, 0.8077937364578247, 0.8056093454360962]}, {"step": 200, "epoch": 0, "train_loss": 0.33325451612472534, "cos_r_negg": [0.47538071870803833, 0.40626096725463867, 0.38132086396217346], "cos_apical_negg": [0.18437808752059937, 0.1742359697818756, 0.20737165212631226], "cos_Ac_negg": [0.4757429361343384, 0.40684956312179565, 0.3820774555206299], "r_norm": [0.7470494508743286, 0.7296936511993408, 0.7287291288375854]}, {"step": 300, "epoch": 0, "train_loss": 0.17068177461624146, "cos_r_negg": [0.49820953607559204, 0.4700973331928253, 0.4245585501194], "cos_apical_negg": [0.16765926778316498, 0.16467534005641937, 0.1947571039199829], "cos_Ac_negg": [0.49843108654022217, 0.47026216983795166, 0.42471715807914734], "r_norm": [0.5785117149353027, 0.567447304725647, 0.5670515298843384]}, {"step": 400, "epoch": 0, "train_loss": 0.12307089567184448, "cos_r_negg": [0.5288668870925903, 0.4906129837036133, 0.459243506193161], "cos_apical_negg": [0.1769542396068573, 0.16480687260627747, 0.2023644745349884], "cos_Ac_negg": [0.5291136503219604, 0.49063006043434143, 0.45918673276901245], "r_norm": [0.5669718980789185, 0.5524458885192871, 0.5533506274223328]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9433, "test_loss": 0.1909028839111328}, {"step": 500, "epoch": 1, "train_loss": 0.23578664660453796, "cos_r_negg": [0.5039648413658142, 0.4705522656440735, 0.46879899501800537], "cos_apical_negg": [0.13089649379253387, 0.12007781863212585, 0.1774221956729889], "cos_Ac_negg": [0.5044163465499878, 0.47066530585289, 0.46881747245788574], "r_norm": [0.4703829884529114, 0.45642536878585815, 0.4560773968696594], "ldr": {"dL_sdil": -0.00954069197177887, "dL_bp": -0.008603498339653015, "ratio": 1.1089316921009196}}, {"step": 600, "epoch": 1, "train_loss": 0.08584421873092651, "cos_r_negg": [0.5099847912788391, 0.4956558346748352, 0.46174073219299316], "cos_apical_negg": [0.13060176372528076, 0.1391550749540329, 0.18097913265228271], "cos_Ac_negg": [0.5103371739387512, 0.4957599937915802, 0.4618017375469208], "r_norm": [0.454019695520401, 0.4426572620868683, 0.441994309425354]}, {"step": 700, "epoch": 1, "train_loss": 0.24514204263687134, "cos_r_negg": [0.5013740062713623, 0.5051541328430176, 0.48172688484191895], "cos_apical_negg": [0.12025074660778046, 0.13877958059310913, 0.1848120093345642], "cos_Ac_negg": [0.5026636123657227, 0.5054739713668823, 0.4818348288536072], "r_norm": [0.5058645009994507, 0.48789969086647034, 0.4897555112838745]}, {"step": 800, "epoch": 1, "train_loss": 0.1141829863190651, "cos_r_negg": [0.5200186967849731, 0.533027172088623, 0.48812541365623474], "cos_apical_negg": [0.10735809803009033, 0.12732172012329102, 0.17967012524604797], "cos_Ac_negg": [0.520397424697876, 0.5329480171203613, 0.4880942106246948], "r_norm": [0.3883897662162781, 0.37802743911743164, 0.37907230854034424]}, {"step": 900, "epoch": 1, "train_loss": 0.18952405452728271, "cos_r_negg": [0.5319833755493164, 0.5358093976974487, 0.4649391174316406], "cos_apical_negg": [0.11764852702617645, 0.1416984498500824, 0.18025359511375427], "cos_Ac_negg": [0.5320981740951538, 0.5356863737106323, 0.4647519886493683], "r_norm": [0.43446850776672363, 0.423671156167984, 0.4214130938053131]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9542, "test_loss": 0.15521985359191895}, {"step": 1000, "epoch": 2, "train_loss": 0.024264752864837646, "cos_r_negg": [0.5205782651901245, 0.5371448993682861, 0.4922335147857666], "cos_apical_negg": [0.0753626674413681, 0.10885260999202728, 0.1598813384771347], "cos_Ac_negg": [0.5206393003463745, 0.5371958017349243, 0.4920315742492676], "r_norm": [0.3322855830192566, 0.322062611579895, 0.32240360975265503], "ldr": {"dL_sdil": -0.007700175046920776, "dL_bp": -0.00669647753238678, "ratio": 1.1498843996234922}}, {"step": 1100, "epoch": 2, "train_loss": 0.09112375229597092, "cos_r_negg": [0.5297616124153137, 0.5311806201934814, 0.47883129119873047], "cos_apical_negg": [0.10208173841238022, 0.11285039037466049, 0.16529834270477295], "cos_Ac_negg": [0.5300719738006592, 0.5313221216201782, 0.4783281683921814], "r_norm": [0.45405858755111694, 0.4363807737827301, 0.44002217054367065]}, {"step": 1200, "epoch": 2, "train_loss": 0.17958039045333862, "cos_r_negg": [0.5119986534118652, 0.5457755327224731, 0.481785386800766], "cos_apical_negg": [0.08440572023391724, 0.10281778872013092, 0.15573155879974365], "cos_Ac_negg": [0.5122300386428833, 0.5457471609115601, 0.4812276363372803], "r_norm": [0.3157377243041992, 0.30727627873420715, 0.30766355991363525]}, {"step": 1300, "epoch": 2, "train_loss": 0.09322493523359299, "cos_r_negg": [0.5049905776977539, 0.5388824939727783, 0.46652308106422424], "cos_apical_negg": [0.0809440165758133, 0.12186141312122345, 0.16658136248588562], "cos_Ac_negg": [0.5050003528594971, 0.5386176109313965, 0.46570855379104614], "r_norm": [0.3353491425514221, 0.32742026448249817, 0.32679492235183716]}, {"step": 1400, "epoch": 2, "train_loss": 0.1470850557088852, "cos_r_negg": [0.4831010103225708, 0.5044246912002563, 0.4572910666465759], "cos_apical_negg": [0.06598492711782455, 0.10995836555957794, 0.16154107451438904], "cos_Ac_negg": [0.4833557903766632, 0.5039969682693481, 0.45643818378448486], "r_norm": [0.2957005202770233, 0.28562095761299133, 0.28597012162208557]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9652, "test_loss": 0.11602793197631836}, {"step": 1500, "epoch": 3, "train_loss": 0.07360994815826416, "cos_r_negg": [0.48249030113220215, 0.5153943300247192, 0.46486377716064453], "cos_apical_negg": [0.07952485978603363, 0.11011232435703278, 0.15342316031455994], "cos_Ac_negg": [0.4827117323875427, 0.5150834321975708, 0.46398216485977173], "r_norm": [0.3044481873512268, 0.2948354482650757, 0.29513657093048096], "ldr": {"dL_sdil": -0.008043169975280762, "dL_bp": -0.00858684629201889, "ratio": 0.9366849832582361}}, {"step": 1600, "epoch": 3, "train_loss": 0.08401516079902649, "cos_r_negg": [0.4855751097202301, 0.5277508497238159, 0.48270171880722046], "cos_apical_negg": [0.06901080906391144, 0.10205385833978653, 0.14363208413124084], "cos_Ac_negg": [0.4855998754501343, 0.5274589657783508, 0.4819275140762329], "r_norm": [0.26068419218063354, 0.2508631944656372, 0.25316059589385986]}, {"step": 1700, "epoch": 3, "train_loss": 0.08663123846054077, "cos_r_negg": [0.4972115159034729, 0.529632568359375, 0.4805557429790497], "cos_apical_negg": [0.07249507308006287, 0.11569879949092865, 0.15890061855316162], "cos_Ac_negg": [0.4972943961620331, 0.5294667482376099, 0.4798460006713867], "r_norm": [0.2743309736251831, 0.26602956652641296, 0.26802799105644226]}, {"step": 1800, "epoch": 3, "train_loss": 0.024224739521741867, "cos_r_negg": [0.48034945130348206, 0.5183444023132324, 0.45742934942245483], "cos_apical_negg": [0.05509719252586365, 0.10797683894634247, 0.13432015478610992], "cos_Ac_negg": [0.48028767108917236, 0.5179896354675293, 0.4565538465976715], "r_norm": [0.27073994278907776, 0.26259320974349976, 0.2608429193496704]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9666, "test_loss": 0.11291602745056152}, {"step": 1900, "epoch": 4, "train_loss": 0.036836519837379456, "cos_r_negg": [0.50064617395401, 0.5486429929733276, 0.5009961128234863], "cos_apical_negg": [0.06748092174530029, 0.12254730612039566, 0.1470053642988205], "cos_Ac_negg": [0.5007407665252686, 0.5483829975128174, 0.5001752376556396], "r_norm": [0.29728272557258606, 0.28962016105651855, 0.2890889346599579]}, {"step": 2000, "epoch": 4, "train_loss": 0.11035207659006119, "cos_r_negg": [0.4658629298210144, 0.5325191020965576, 0.47461599111557007], "cos_apical_negg": [0.03910278156399727, 0.09647335112094879, 0.13209731876850128], "cos_Ac_negg": [0.4660146236419678, 0.532520592212677, 0.4736754894256592], "r_norm": [0.2169564962387085, 0.21035748720169067, 0.21062403917312622], "ldr": {"dL_sdil": -0.005091726779937744, "dL_bp": -0.006505265831947327, "ratio": 0.7827084874736864}}, {"step": 2100, "epoch": 4, "train_loss": 0.04881729558110237, "cos_r_negg": [0.4404827952384949, 0.5042537450790405, 0.4767466187477112], "cos_apical_negg": [0.06356577575206757, 0.11879993975162506, 0.16410501301288605], "cos_Ac_negg": [0.44049811363220215, 0.503831148147583, 0.4753091335296631], "r_norm": [0.2721758484840393, 0.2628178596496582, 0.2637660503387451]}, {"step": 2200, "epoch": 4, "train_loss": 0.014741148799657822, "cos_r_negg": [0.4503231644630432, 0.5060051679611206, 0.4661533832550049], "cos_apical_negg": [0.06367862224578857, 0.0858410894870758, 0.14047116041183472], "cos_Ac_negg": [0.45045188069343567, 0.5057669878005981, 0.46503353118896484], "r_norm": [0.21717336773872375, 0.2100096046924591, 0.20898042619228363]}, {"step": 2300, "epoch": 4, "train_loss": 0.11582493782043457, "cos_r_negg": [0.41866520047187805, 0.46615898609161377, 0.43160831928253174], "cos_apical_negg": [0.055221423506736755, 0.0981120616197586, 0.1351119428873062], "cos_Ac_negg": [0.4186370074748993, 0.4656684398651123, 0.43063122034072876], "r_norm": [0.21510441601276398, 0.20846328139305115, 0.2084541618824005]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9687, "test_loss": 0.11309259471893311}, {"step": 2400, "epoch": 5, "train_loss": 0.08978386223316193, "cos_r_negg": [0.4204558730125427, 0.4770914912223816, 0.4370455741882324], "cos_apical_negg": [0.041964802891016006, 0.09495782852172852, 0.1311182826757431], "cos_Ac_negg": [0.420477032661438, 0.4764206111431122, 0.43577802181243896], "r_norm": [0.2225853055715561, 0.21526116132736206, 0.21467289328575134]}, {"step": 2500, "epoch": 5, "train_loss": 0.032590627670288086, "cos_r_negg": [0.37645357847213745, 0.4301737844944, 0.3948667049407959], "cos_apical_negg": [0.041951730847358704, 0.0894026979804039, 0.13183176517486572], "cos_Ac_negg": [0.376258909702301, 0.42983150482177734, 0.3935735821723938], "r_norm": [0.1877080649137497, 0.18147850036621094, 0.1814000904560089], "ldr": {"dL_sdil": -0.005059204995632172, "dL_bp": -0.005994416773319244, "ratio": 0.8439861936444528}}, {"step": 2600, "epoch": 5, "train_loss": 0.04170890152454376, "cos_r_negg": [0.3668670058250427, 0.4246392250061035, 0.39079034328460693], "cos_apical_negg": [0.05386599898338318, 0.08447977900505066, 0.13276056945323944], "cos_Ac_negg": [0.3666914105415344, 0.4242306649684906, 0.3895471394062042], "r_norm": [0.22902095317840576, 0.22174498438835144, 0.2195884883403778]}, {"step": 2700, "epoch": 5, "train_loss": 0.027166618034243584, "cos_r_negg": [0.40010547637939453, 0.4535364806652069, 0.42476123571395874], "cos_apical_negg": [0.04932790249586105, 0.08109010756015778, 0.12461468577384949], "cos_Ac_negg": [0.3998187184333801, 0.4530128836631775, 0.42341580986976624], "r_norm": [0.18988904356956482, 0.18261617422103882, 0.18350571393966675]}, {"step": 2800, "epoch": 5, "train_loss": 0.11552451550960541, "cos_r_negg": [0.3934221565723419, 0.4440481662750244, 0.41737568378448486], "cos_apical_negg": [0.05744766816496849, 0.10390763729810715, 0.14613085985183716], "cos_Ac_negg": [0.39355671405792236, 0.44348573684692383, 0.41604793071746826], "r_norm": [0.2742878794670105, 0.2649727761745453, 0.2640365660190582]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9673, "test_loss": 0.11689610671997071}, {"step": 2900, "epoch": 6, "train_loss": 0.0037757144309580326, "cos_r_negg": [0.3694411516189575, 0.42569172382354736, 0.3944653272628784], "cos_apical_negg": [0.04000677913427353, 0.07870016992092133, 0.138505756855011], "cos_Ac_negg": [0.36947134137153625, 0.42532140016555786, 0.3929949998855591], "r_norm": [0.23445159196853638, 0.22432029247283936, 0.22750048339366913]}, {"step": 3000, "epoch": 6, "train_loss": 0.02827419713139534, "cos_r_negg": [0.3648776412010193, 0.41716763377189636, 0.3857506215572357], "cos_apical_negg": [0.055921055376529694, 0.07587356120347977, 0.1285562515258789], "cos_Ac_negg": [0.3648226261138916, 0.41677188873291016, 0.3843785524368286], "r_norm": [0.21666641533374786, 0.20692607760429382, 0.20901024341583252], "ldr": {"dL_sdil": -0.006700620055198669, "dL_bp": -0.0091777965426445, "ratio": 0.7300902808276839}}, {"step": 3100, "epoch": 6, "train_loss": 0.020287038758397102, "cos_r_negg": [0.35229307413101196, 0.39654719829559326, 0.3670850396156311], "cos_apical_negg": [0.04153112694621086, 0.07681026309728622, 0.12110616266727448], "cos_Ac_negg": [0.352527379989624, 0.3960666358470917, 0.3658161461353302], "r_norm": [0.2427251636981964, 0.2328503429889679, 0.2335037887096405]}, {"step": 3200, "epoch": 6, "train_loss": 0.06322503089904785, "cos_r_negg": [0.33731675148010254, 0.38782185316085815, 0.36453816294670105], "cos_apical_negg": [0.035065554082393646, 0.07769809663295746, 0.1290198415517807], "cos_Ac_negg": [0.3375152349472046, 0.3873109221458435, 0.3632696866989136], "r_norm": [0.23465991020202637, 0.2265673279762268, 0.22736138105392456]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9692, "test_loss": 0.12009874725341797}, {"step": 3300, "epoch": 7, "train_loss": 0.004691869020462036, "cos_r_negg": [0.32288289070129395, 0.3667840361595154, 0.3451842665672302], "cos_apical_negg": [0.03087460622191429, 0.07438303530216217, 0.11760503053665161], "cos_Ac_negg": [0.32292357087135315, 0.36617881059646606, 0.34379759430885315], "r_norm": [0.20154176652431488, 0.1956992894411087, 0.19322997331619263]}, {"step": 3400, "epoch": 7, "train_loss": 0.05418068915605545, "cos_r_negg": [0.2925570607185364, 0.33336225152015686, 0.3140586018562317], "cos_apical_negg": [0.03491266444325447, 0.08083079755306244, 0.12677988409996033], "cos_Ac_negg": [0.2926652431488037, 0.33279937505722046, 0.31269651651382446], "r_norm": [0.20136584341526031, 0.19460076093673706, 0.19472603499889374]}, {"step": 3500, "epoch": 7, "train_loss": 0.011063420213758945, "cos_r_negg": [0.28856903314590454, 0.3280424475669861, 0.31342777609825134], "cos_apical_negg": [0.027785707265138626, 0.0772261768579483, 0.11969386041164398], "cos_Ac_negg": [0.2886590361595154, 0.32759934663772583, 0.31211069226264954], "r_norm": [0.2028031349182129, 0.19627855718135834, 0.19565483927726746], "ldr": {"dL_sdil": -0.008338302373886108, "dL_bp": -0.013276100158691406, "ratio": 0.628068655269018}}, {"step": 3600, "epoch": 7, "train_loss": 0.008701559156179428, "cos_r_negg": [0.29482460021972656, 0.33869469165802, 0.32127347588539124], "cos_apical_negg": [0.041596755385398865, 0.0818106159567833, 0.12602707743644714], "cos_Ac_negg": [0.2949414849281311, 0.33830952644348145, 0.3200538456439972], "r_norm": [0.23024503886699677, 0.22326207160949707, 0.22281455993652344]}, {"step": 3700, "epoch": 7, "train_loss": 0.003036912064999342, "cos_r_negg": [0.27759861946105957, 0.33036568760871887, 0.30939558148384094], "cos_apical_negg": [0.03221322223544121, 0.06793379783630371, 0.11873779445886612], "cos_Ac_negg": [0.2778462767601013, 0.33007898926734924, 0.30816054344177246], "r_norm": [0.19145478308200836, 0.18450607359409332, 0.1833382099866867]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9724, "test_loss": 0.11683763999938965}, {"step": 3800, "epoch": 8, "train_loss": 0.004788301885128021, "cos_r_negg": [0.26186060905456543, 0.3063490092754364, 0.28885629773139954], "cos_apical_negg": [0.03315182775259018, 0.06766022741794586, 0.1137072965502739], "cos_Ac_negg": [0.26219868659973145, 0.3061567544937134, 0.28769949078559875], "r_norm": [0.18467822670936584, 0.17843174934387207, 0.17825502157211304]}, {"step": 3900, "epoch": 8, "train_loss": 0.014764433726668358, "cos_r_negg": [0.23690886795520782, 0.2811741232872009, 0.26531487703323364], "cos_apical_negg": [0.03204338625073433, 0.06984101235866547, 0.11686915159225464], "cos_Ac_negg": [0.23701108992099762, 0.2810598909854889, 0.26432308554649353], "r_norm": [0.19513821601867676, 0.18771901726722717, 0.18832020461559296]}, {"step": 4000, "epoch": 8, "train_loss": 0.013668123632669449, "cos_r_negg": [0.22913989424705505, 0.2737671136856079, 0.25768041610717773], "cos_apical_negg": [0.02369956113398075, 0.07034076750278473, 0.11756635457277298], "cos_Ac_negg": [0.22939442098140717, 0.2735897898674011, 0.2565460205078125], "r_norm": [0.1800396591424942, 0.1737038940191269, 0.17378592491149902], "ldr": {"dL_sdil": -0.0061127543449401855, "dL_bp": -0.010247230529785156, "ratio": 0.5965274546300605}}, {"step": 4100, "epoch": 8, "train_loss": 0.010922418907284737, "cos_r_negg": [0.22424617409706116, 0.26542943716049194, 0.25373709201812744], "cos_apical_negg": [0.025014229118824005, 0.06453858315944672, 0.1126052513718605], "cos_Ac_negg": [0.22420212626457214, 0.2649170756340027, 0.25251391530036926], "r_norm": [0.17920726537704468, 0.17361032962799072, 0.17222094535827637]}, {"step": 4200, "epoch": 8, "train_loss": 0.015913143754005432, "cos_r_negg": [0.22936886548995972, 0.27066126465797424, 0.2599010467529297], "cos_apical_negg": [0.027789544314146042, 0.07163587212562561, 0.11642490327358246], "cos_Ac_negg": [0.22943341732025146, 0.2704053521156311, 0.2588973939418793], "r_norm": [0.21224844455718994, 0.20578020811080933, 0.20467448234558105]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9723, "test_loss": 0.12563416786193848}, {"step": 4300, "epoch": 9, "train_loss": 0.0033008893951773643, "cos_r_negg": [0.21392591297626495, 0.2511399984359741, 0.24229983985424042], "cos_apical_negg": [0.030487868934869766, 0.06699858605861664, 0.11371258646249771], "cos_Ac_negg": [0.2138318121433258, 0.2509313225746155, 0.24132709205150604], "r_norm": [0.21782080829143524, 0.21160924434661865, 0.21103334426879883]}, {"step": 4400, "epoch": 9, "train_loss": 0.0038509261794388294, "cos_r_negg": [0.2087613195180893, 0.2530443072319031, 0.24557706713676453], "cos_apical_negg": [0.03508983552455902, 0.07051075994968414, 0.11215881258249283], "cos_Ac_negg": [0.2089308500289917, 0.2528691291809082, 0.24463456869125366], "r_norm": [0.20658817887306213, 0.1997489333152771, 0.19995152950286865]}, {"step": 4500, "epoch": 9, "train_loss": 0.04732319340109825, "cos_r_negg": [0.20101208984851837, 0.23820745944976807, 0.22872284054756165], "cos_apical_negg": [0.033846884965896606, 0.06669357419013977, 0.10973863303661346], "cos_Ac_negg": [0.20120809972286224, 0.23792393505573273, 0.22767271101474762], "r_norm": [0.20002463459968567, 0.19208872318267822, 0.19229763746261597], "ldr": {"dL_sdil": -0.009978234767913818, "dL_bp": -0.017390698194503784, "ratio": 0.5737684971766903}}, {"step": 4600, "epoch": 9, "train_loss": 0.001096121734008193, "cos_r_negg": [0.21702511608600616, 0.2599443197250366, 0.24847272038459778], "cos_apical_negg": [0.027071991935372353, 0.0694192424416542, 0.11013548821210861], "cos_Ac_negg": [0.21716487407684326, 0.25956571102142334, 0.24733448028564453], "r_norm": [0.21108755469322205, 0.202933669090271, 0.2025962918996811]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9717, "test_loss": 0.12970950088500977}, {"step": 4700, "epoch": 10, "train_loss": 0.0036608714144676924, "cos_r_negg": [0.18710611760616302, 0.22294868528842926, 0.21843338012695312], "cos_apical_negg": [0.023753192275762558, 0.06152021139860153, 0.10615641623735428], "cos_Ac_negg": [0.18703237175941467, 0.22290509939193726, 0.2174932360649109], "r_norm": [0.19547519087791443, 0.1901041567325592, 0.18925872445106506]}, {"step": 4800, "epoch": 10, "train_loss": 0.003912936896085739, "cos_r_negg": [0.18494813144207, 0.21798130869865417, 0.21301724016666412], "cos_apical_negg": [0.028122922405600548, 0.06218917667865753, 0.10311560332775116], "cos_Ac_negg": [0.18500551581382751, 0.21800583600997925, 0.21219076216220856], "r_norm": [0.19273266196250916, 0.18663793802261353, 0.18621589243412018]}, {"step": 4900, "epoch": 10, "train_loss": 0.0015825561713427305, "cos_r_negg": [0.1864510029554367, 0.22116166353225708, 0.21488024294376373], "cos_apical_negg": [0.027721472084522247, 0.06270502507686615, 0.10014031082391739], "cos_Ac_negg": [0.18659408390522003, 0.22118352353572845, 0.21406729519367218], "r_norm": [0.19535258412361145, 0.18831703066825867, 0.1883232295513153]}, {"step": 5000, "epoch": 10, "train_loss": 0.00045510963536798954, "cos_r_negg": [0.18957404792308807, 0.22415143251419067, 0.21770745515823364], "cos_apical_negg": [0.028135128319263458, 0.059308186173439026, 0.10442541539669037], "cos_Ac_negg": [0.18959182500839233, 0.22407618165016174, 0.2168348729610443], "r_norm": [0.20160190761089325, 0.19463421404361725, 0.19437533617019653], "ldr": {"dL_sdil": -0.010283231735229492, "dL_bp": -0.018801160156726837, "ratio": 0.5469466591161541}}, {"step": 5100, "epoch": 10, "train_loss": 0.00398306455463171, "cos_r_negg": [0.18066635727882385, 0.21527476608753204, 0.20831051468849182], "cos_apical_negg": [0.025609023869037628, 0.06073661148548126, 0.10361998528242111], "cos_Ac_negg": [0.1806965321302414, 0.215121328830719, 0.2074241042137146], "r_norm": [0.2056700885295868, 0.20005041360855103, 0.19889876246452332]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.973, "test_loss": 0.13707104358673095}, {"step": 5200, "epoch": 11, "train_loss": 0.0010212808847427368, "cos_r_negg": [0.17511871457099915, 0.2060219794511795, 0.20122885704040527], "cos_apical_negg": [0.02258995734155178, 0.055186498910188675, 0.0963829755783081], "cos_Ac_negg": [0.17518267035484314, 0.20604665577411652, 0.20043078064918518], "r_norm": [0.19949084520339966, 0.19359317421913147, 0.19263364374637604]}, {"step": 5300, "epoch": 11, "train_loss": 0.0023902258835732937, "cos_r_negg": [0.17802681028842926, 0.21074262261390686, 0.20774811506271362], "cos_apical_negg": [0.024833615869283676, 0.05695783346891403, 0.10171078145503998], "cos_Ac_negg": [0.17808008193969727, 0.2107875645160675, 0.2069690078496933], "r_norm": [0.20352932810783386, 0.19760927557945251, 0.19624817371368408]}, {"step": 5400, "epoch": 11, "train_loss": 0.0010215830989181995, "cos_r_negg": [0.17006847262382507, 0.1999242901802063, 0.19609680771827698], "cos_apical_negg": [0.023248709738254547, 0.05559142678976059, 0.09681231528520584], "cos_Ac_negg": [0.1700647473335266, 0.19989797472953796, 0.1953248232603073], "r_norm": [0.19342386722564697, 0.18735268712043762, 0.18649378418922424]}, {"step": 5500, "epoch": 11, "train_loss": 0.002317067002877593, "cos_r_negg": [0.16678230464458466, 0.19703422486782074, 0.19261083006858826], "cos_apical_negg": [0.019725129008293152, 0.05744431912899017, 0.09821569919586182], "cos_Ac_negg": [0.16692307591438293, 0.1971496194601059, 0.19187787175178528], "r_norm": [0.20514193177223206, 0.19878599047660828, 0.19838057458400726], "ldr": {"dL_sdil": -0.01085682213306427, "dL_bp": -0.020643025636672974, "ratio": 0.5259317274584395}}, {"step": 5600, "epoch": 11, "train_loss": 0.0012320679379627109, "cos_r_negg": [0.1612672358751297, 0.191120907664299, 0.188379168510437], "cos_apical_negg": [0.020308878272771835, 0.056304868310689926, 0.09708579629659653], "cos_Ac_negg": [0.16140258312225342, 0.191266268491745, 0.18763467669487], "r_norm": [0.19809508323669434, 0.19155529141426086, 0.19131436944007874]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9734, "test_loss": 0.13729743576049805}, {"step": 5700, "epoch": 12, "train_loss": 0.0008859423105604947, "cos_r_negg": [0.15953657031059265, 0.18740582466125488, 0.18432173132896423], "cos_apical_negg": [0.02143375761806965, 0.05570517107844353, 0.09517017751932144], "cos_Ac_negg": [0.15969060361385345, 0.1875518560409546, 0.18358762562274933], "r_norm": [0.19843733310699463, 0.19255031645298004, 0.19181010127067566]}, {"step": 5800, "epoch": 12, "train_loss": 0.0003311795589979738, "cos_r_negg": [0.15922468900680542, 0.18801870942115784, 0.18526813387870789], "cos_apical_negg": [0.023380853235721588, 0.056249793618917465, 0.09497200697660446], "cos_Ac_negg": [0.1592942178249359, 0.18813161551952362, 0.1845625340938568], "r_norm": [0.19794294238090515, 0.191816508769989, 0.19135384261608124]}, {"step": 5900, "epoch": 12, "train_loss": 0.00156401249114424, "cos_r_negg": [0.15981990098953247, 0.18862323462963104, 0.18633194267749786], "cos_apical_negg": [0.020965633913874626, 0.05583123490214348, 0.09419971704483032], "cos_Ac_negg": [0.15990784764289856, 0.1887335628271103, 0.18565072119235992], "r_norm": [0.19851912558078766, 0.19241592288017273, 0.1919378638267517]}, {"step": 6000, "epoch": 12, "train_loss": 0.00030895404051989317, "cos_r_negg": [0.1553572416305542, 0.18239045143127441, 0.17960326373577118], "cos_apical_negg": [0.01996655762195587, 0.054028622806072235, 0.09105212986469269], "cos_Ac_negg": [0.15546205639839172, 0.18260279297828674, 0.17893171310424805], "r_norm": [0.19863979518413544, 0.19224649667739868, 0.19180689752101898], "ldr": {"dL_sdil": -0.010728120803833008, "dL_bp": -0.020957522094249725, "ratio": 0.5118983415876519}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9724, "test_loss": 0.14106293849945067}, {"step": 6100, "epoch": 13, "train_loss": 0.0011298148892819881, "cos_r_negg": [0.15571193397045135, 0.183889240026474, 0.18201139569282532], "cos_apical_negg": [0.02182769775390625, 0.05595606565475464, 0.09365227818489075], "cos_Ac_negg": [0.1558626890182495, 0.1840338110923767, 0.18130674958229065], "r_norm": [0.1986820101737976, 0.1925334632396698, 0.19183975458145142]}, {"step": 6200, "epoch": 13, "train_loss": 0.0006148054962977767, "cos_r_negg": [0.1534225344657898, 0.17917712032794952, 0.17740431427955627], "cos_apical_negg": [0.018896566703915596, 0.053799763321876526, 0.08818738162517548], "cos_Ac_negg": [0.15356265008449554, 0.17936858534812927, 0.17675183713436127], "r_norm": [0.19784966111183167, 0.19142046570777893, 0.19122035801410675]}, {"step": 6300, "epoch": 13, "train_loss": 0.000752944964915514, "cos_r_negg": [0.15010854601860046, 0.1755460947751999, 0.1741248369216919], "cos_apical_negg": [0.02037227712571621, 0.05345896631479263, 0.08759614825248718], "cos_Ac_negg": [0.15027117729187012, 0.1757322996854782, 0.1734912395477295], "r_norm": [0.20273363590240479, 0.19651848077774048, 0.19604083895683289]}, {"step": 6400, "epoch": 13, "train_loss": 0.000505276839248836, "cos_r_negg": [0.14961662888526917, 0.17585255205631256, 0.17388848960399628], "cos_apical_negg": [0.02023317851126194, 0.05379866808652878, 0.0907919630408287], "cos_Ac_negg": [0.14980493485927582, 0.17601051926612854, 0.17325332760810852], "r_norm": [0.1988239884376526, 0.19270282983779907, 0.19201262295246124]}, {"step": 6500, "epoch": 13, "train_loss": 0.00012816628441214561, "cos_r_negg": [0.14679403603076935, 0.17271862924098969, 0.17046913504600525], "cos_apical_negg": [0.019949106499552727, 0.05523088574409485, 0.08674812316894531], "cos_Ac_negg": [0.14698968827724457, 0.17292772233486176, 0.16982415318489075], "r_norm": [0.19890861213207245, 0.19264927506446838, 0.19208171963691711], "ldr": {"dL_sdil": -0.01122257113456726, "dL_bp": -0.02264733612537384, "ratio": 0.49553603445632655}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9722, "test_loss": 0.14485032234191894}, {"step": 6600, "epoch": 14, "train_loss": 0.00019565958064049482, "cos_r_negg": [0.14653591811656952, 0.17161259055137634, 0.16988086700439453], "cos_apical_negg": [0.018932346254587173, 0.05328367277979851, 0.08595643192529678], "cos_Ac_negg": [0.14674195647239685, 0.1718364804983139, 0.16925981640815735], "r_norm": [0.20191246271133423, 0.19600293040275574, 0.19518572092056274]}, {"step": 6700, "epoch": 14, "train_loss": 0.00022406170319300145, "cos_r_negg": [0.14670738577842712, 0.1720474511384964, 0.17007070779800415], "cos_apical_negg": [0.02101265639066696, 0.05436839163303375, 0.08754845708608627], "cos_Ac_negg": [0.1469302475452423, 0.1723116934299469, 0.16945433616638184], "r_norm": [0.20270946621894836, 0.19609719514846802, 0.19590112566947937]}, {"step": 6800, "epoch": 14, "train_loss": 0.0005333982408046722, "cos_r_negg": [0.14481744170188904, 0.16912712156772614, 0.16751673817634583], "cos_apical_negg": [0.01939425989985466, 0.05289234593510628, 0.08642574399709702], "cos_Ac_negg": [0.145021453499794, 0.16937115788459778, 0.16693395376205444], "r_norm": [0.20019349455833435, 0.1938047707080841, 0.19334977865219116]}, {"step": 6900, "epoch": 14, "train_loss": 0.0007012758869677782, "cos_r_negg": [0.1437663733959198, 0.16749265789985657, 0.1661156266927719], "cos_apical_negg": [0.020346809178590775, 0.051871880888938904, 0.08374530076980591], "cos_Ac_negg": [0.14397886395454407, 0.1677415668964386, 0.165548175573349], "r_norm": [0.20070359110832214, 0.19453108310699463, 0.1939391940832138]}, {"step": 7000, "epoch": 14, "train_loss": 0.001409738790243864, "cos_r_negg": [0.14288873970508575, 0.1668173372745514, 0.16489183902740479], "cos_apical_negg": [0.019075414165854454, 0.05251060798764229, 0.08234767615795135], "cos_Ac_negg": [0.1431255042552948, 0.1670810580253601, 0.1643112301826477], "r_norm": [0.2023114114999771, 0.1960645318031311, 0.19565320014953613], "ldr": {"dL_sdil": -0.011931747198104858, "dL_bp": -0.024229615926742554, "ratio": 0.4924447516700266}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9724, "test_loss": 0.1476565719604492}], "final": {"test_acc": 0.9724, "test_loss": 0.1476565719604492, "wall_s": 39.97539758682251, "cos_r_negg": [0.14380186796188354, 0.16867755353450775, 0.1670524775981903], "cos_apical_negg": [0.020168934017419815, 0.053159285336732864, 0.0830625593662262], "cos_Ac_negg": [0.144031822681427, 0.16894739866256714, 0.16645359992980957]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p1_res0_s0.json b/results/diag_nuis_low_rho0p1_res0_s0.json new file mode 100644 index 0000000..eee2415 --- /dev/null +++ b/results/diag_nuis_low_rho0p1_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.1, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p1_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.018149716779589653, -0.026159491389989853, 0.03341463953256607], "cos_apical_negg": [-0.01817035675048828, -0.026294099166989326, 0.03202316164970398], "cos_Ac_negg": [-0.018149685114622116, -0.026159506291151047, 0.03341459482908249], "r_norm": [4.897443771362305, 4.8264689445495605, 4.823726177215576], "ldr": {"dL_sdil": -0.007371425628662109, "dL_bp": -0.22916293144226074, "ratio": 0.032166745216031564}}, {"step": 100, "epoch": 0, "train_loss": 0.2762371301651001, "cos_r_negg": [0.44541698694229126, 0.44750165939331055, 0.4498120844364166], "cos_apical_negg": [0.1863493025302887, 0.22645536065101624, 0.29871681332588196], "cos_Ac_negg": [0.44527506828308105, 0.4474462866783142, 0.4496145248413086], "r_norm": [0.8935065865516663, 0.8772377371788025, 0.8749199509620667]}, {"step": 200, "epoch": 0, "train_loss": 0.37479275465011597, "cos_r_negg": [0.44363704323768616, 0.4628668427467346, 0.5308301448822021], "cos_apical_negg": [0.17599457502365112, 0.20544520020484924, 0.33132797479629517], "cos_Ac_negg": [0.4437377452850342, 0.4631724953651428, 0.5308259129524231], "r_norm": [0.759809136390686, 0.744493842124939, 0.7407262325286865]}, {"step": 300, "epoch": 0, "train_loss": 0.2960105538368225, "cos_r_negg": [0.49565568566322327, 0.5042164325714111, 0.4768705368041992], "cos_apical_negg": [0.16109946370124817, 0.23191282153129578, 0.2783460319042206], "cos_Ac_negg": [0.49555158615112305, 0.5041651725769043, 0.47681987285614014], "r_norm": [0.7143169641494751, 0.7000225782394409, 0.6968193650245667]}, {"step": 400, "epoch": 0, "train_loss": 0.156812384724617, "cos_r_negg": [0.46017366647720337, 0.44548624753952026, 0.4824177622795105], "cos_apical_negg": [0.140870600938797, 0.16781483590602875, 0.2878004312515259], "cos_Ac_negg": [0.4601784944534302, 0.44544240832328796, 0.48233509063720703], "r_norm": [0.6665322780609131, 0.6561061143875122, 0.6546273231506348]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9386, "test_loss": 0.24025605010986328}, {"step": 500, "epoch": 1, "train_loss": 0.1977778971195221, "cos_r_negg": [0.46847325563430786, 0.48639601469039917, 0.4975510835647583], "cos_apical_negg": [0.12961715459823608, 0.19412121176719666, 0.29938745498657227], "cos_Ac_negg": [0.468395471572876, 0.4863022565841675, 0.4973657727241516], "r_norm": [0.5680610537528992, 0.555564284324646, 0.5551686882972717], "ldr": {"dL_sdil": -0.009178996086120605, "dL_bp": -0.007929801940917969, "ratio": 1.1575315694527961}}, {"step": 600, "epoch": 1, "train_loss": 0.12994696199893951, "cos_r_negg": [0.40892812609672546, 0.44459545612335205, 0.49697914719581604], "cos_apical_negg": [0.1473712921142578, 0.18131253123283386, 0.28280460834503174], "cos_Ac_negg": [0.4088168442249298, 0.44452303647994995, 0.4969891905784607], "r_norm": [0.5339714288711548, 0.5249640345573425, 0.519861102104187]}, {"step": 700, "epoch": 1, "train_loss": 0.16491901874542236, "cos_r_negg": [0.4256364703178406, 0.440668523311615, 0.5330575704574585], "cos_apical_negg": [0.11386850476264954, 0.18758568167686462, 0.35666579008102417], "cos_Ac_negg": [0.42566782236099243, 0.4405771493911743, 0.5329521894454956], "r_norm": [0.5536546111106873, 0.5378513336181641, 0.5388550162315369]}, {"step": 800, "epoch": 1, "train_loss": 0.17110717296600342, "cos_r_negg": [0.4351373016834259, 0.43780046701431274, 0.576678454875946], "cos_apical_negg": [0.10322420299053192, 0.1988857090473175, 0.3974878191947937], "cos_Ac_negg": [0.43509441614151, 0.43770116567611694, 0.5765353441238403], "r_norm": [0.48688948154449463, 0.475961834192276, 0.4741421639919281]}, {"step": 900, "epoch": 1, "train_loss": 0.29496675729751587, "cos_r_negg": [0.48740899562835693, 0.47181832790374756, 0.5963932275772095], "cos_apical_negg": [0.10475624352693558, 0.2022893726825714, 0.4589501619338989], "cos_Ac_negg": [0.48739027976989746, 0.4717230498790741, 0.5962759852409363], "r_norm": [0.48102694749832153, 0.46774303913116455, 0.46749189496040344]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9467, "test_loss": 0.21232797012329102}, {"step": 1000, "epoch": 2, "train_loss": 0.05081885680556297, "cos_r_negg": [0.4261878430843353, 0.4687647819519043, 0.5351234078407288], "cos_apical_negg": [0.08882705867290497, 0.14287811517715454, 0.41190072894096375], "cos_Ac_negg": [0.42612844705581665, 0.4686488211154938, 0.5350337028503418], "r_norm": [0.4382684528827667, 0.4267258942127228, 0.4252968430519104], "ldr": {"dL_sdil": -0.0077686309814453125, "dL_bp": -0.007604062557220459, "ratio": 1.021642171271801}}, {"step": 1100, "epoch": 2, "train_loss": 0.17293067276477814, "cos_r_negg": [0.42635345458984375, 0.436636745929718, 0.43410253524780273], "cos_apical_negg": [0.11521389335393906, 0.1451309621334076, 0.29686006903648376], "cos_Ac_negg": [0.42632269859313965, 0.43655574321746826, 0.4340246021747589], "r_norm": [0.4487457871437073, 0.4393031597137451, 0.43677783012390137]}, {"step": 1200, "epoch": 2, "train_loss": 0.20848479866981506, "cos_r_negg": [0.442474901676178, 0.45517247915267944, 0.5386346578598022], "cos_apical_negg": [0.09892383217811584, 0.16177767515182495, 0.4463983178138733], "cos_Ac_negg": [0.4423860013484955, 0.4550597071647644, 0.5385562181472778], "r_norm": [0.39272546768188477, 0.38346344232559204, 0.3823698163032532]}, {"step": 1300, "epoch": 2, "train_loss": 0.16447065770626068, "cos_r_negg": [0.4026065468788147, 0.4427296817302704, 0.4520164430141449], "cos_apical_negg": [0.1049981638789177, 0.16617241501808167, 0.43328726291656494], "cos_Ac_negg": [0.4025520086288452, 0.44268229603767395, 0.45195090770721436], "r_norm": [0.45160338282585144, 0.4402342140674591, 0.43770354986190796]}, {"step": 1400, "epoch": 2, "train_loss": 0.20390990376472473, "cos_r_negg": [0.3993721306324005, 0.4542754590511322, 0.4953177571296692], "cos_apical_negg": [0.06672823429107666, 0.1392410546541214, 0.37507134675979614], "cos_Ac_negg": [0.3993054926395416, 0.4541670083999634, 0.49529051780700684], "r_norm": [0.3890014886856079, 0.37828296422958374, 0.37614405155181885]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9554, "test_loss": 0.169120894241333}, {"step": 1500, "epoch": 3, "train_loss": 0.16671910881996155, "cos_r_negg": [0.4117174744606018, 0.4330917298793793, 0.48378750681877136], "cos_apical_negg": [0.0925174355506897, 0.15199881792068481, 0.3734169602394104], "cos_Ac_negg": [0.4117169976234436, 0.4330543875694275, 0.483761191368103], "r_norm": [0.3958891034126282, 0.38500869274139404, 0.38426539301872253], "ldr": {"dL_sdil": -0.005747601389884949, "dL_bp": -0.006682336330413818, "ratio": 0.8601185432294779}}, {"step": 1600, "epoch": 3, "train_loss": 0.12549501657485962, "cos_r_negg": [0.39289334416389465, 0.44915440678596497, 0.4579179883003235], "cos_apical_negg": [0.06114117428660393, 0.16337698698043823, 0.3305564522743225], "cos_Ac_negg": [0.39274075627326965, 0.4489688277244568, 0.45787423849105835], "r_norm": [0.3441646695137024, 0.334293395280838, 0.33339497447013855]}, {"step": 1700, "epoch": 3, "train_loss": 0.10290870815515518, "cos_r_negg": [0.35971707105636597, 0.4149544835090637, 0.41636937856674194], "cos_apical_negg": [0.10031169652938843, 0.17306800186634064, 0.3904416561126709], "cos_Ac_negg": [0.3596198856830597, 0.4148382544517517, 0.41626545786857605], "r_norm": [0.37432676553726196, 0.3625507652759552, 0.3613091707229614]}, {"step": 1800, "epoch": 3, "train_loss": 0.05377885699272156, "cos_r_negg": [0.3381999135017395, 0.39523470401763916, 0.40547239780426025], "cos_apical_negg": [0.07854337990283966, 0.1191946268081665, 0.3450331389904022], "cos_Ac_negg": [0.33815616369247437, 0.3951526880264282, 0.4054502546787262], "r_norm": [0.34665098786354065, 0.33539432287216187, 0.3357474207878113]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9568, "test_loss": 0.1756272834777832}, {"step": 1900, "epoch": 4, "train_loss": 0.053782038390636444, "cos_r_negg": [0.37742263078689575, 0.43863987922668457, 0.4813166856765747], "cos_apical_negg": [0.07947307825088501, 0.18122676014900208, 0.36060473322868347], "cos_Ac_negg": [0.3773921728134155, 0.4385415315628052, 0.4813358783721924], "r_norm": [0.3479138910770416, 0.34022289514541626, 0.3393150269985199]}, {"step": 2000, "epoch": 4, "train_loss": 0.1693098545074463, "cos_r_negg": [0.302074134349823, 0.36862003803253174, 0.3845616579055786], "cos_apical_negg": [0.05067247152328491, 0.09565144032239914, 0.30579322576522827], "cos_Ac_negg": [0.30201858282089233, 0.36856329441070557, 0.3845508396625519], "r_norm": [0.3478071987628937, 0.3374285101890564, 0.3365751802921295], "ldr": {"dL_sdil": -0.00672970712184906, "dL_bp": -0.007786288857460022, "ratio": 0.8643022683908453}}, {"step": 2100, "epoch": 4, "train_loss": 0.08755938708782196, "cos_r_negg": [0.36371514201164246, 0.3920271098613739, 0.4279513955116272], "cos_apical_negg": [0.10892920196056366, 0.14157618582248688, 0.35677605867385864], "cos_Ac_negg": [0.3636758625507355, 0.39195653796195984, 0.4279196560382843], "r_norm": [0.38288623094558716, 0.37204116582870483, 0.3697913885116577]}, {"step": 2200, "epoch": 4, "train_loss": 0.032282549887895584, "cos_r_negg": [0.32277482748031616, 0.37331444025039673, 0.4008542001247406], "cos_apical_negg": [0.05109245330095291, 0.10842463374137878, 0.3036212921142578], "cos_Ac_negg": [0.3227275311946869, 0.3732089698314667, 0.40080153942108154], "r_norm": [0.3336029350757599, 0.3244991898536682, 0.32251161336898804]}, {"step": 2300, "epoch": 4, "train_loss": 0.31710344552993774, "cos_r_negg": [0.3238556683063507, 0.3977438509464264, 0.4312130808830261], "cos_apical_negg": [0.08798874914646149, 0.1668751835823059, 0.30054429173469543], "cos_Ac_negg": [0.3237970471382141, 0.39765965938568115, 0.4312444031238556], "r_norm": [0.3623567819595337, 0.35056763887405396, 0.3507392406463623]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9556, "test_loss": 0.18188404693603516}, {"step": 2400, "epoch": 5, "train_loss": 0.10923928767442703, "cos_r_negg": [0.3178540766239166, 0.37013620138168335, 0.4150301218032837], "cos_apical_negg": [0.03521845489740372, 0.1222638338804245, 0.2965596914291382], "cos_Ac_negg": [0.3178318738937378, 0.3700489103794098, 0.4150121510028839], "r_norm": [0.3304583728313446, 0.3197857141494751, 0.3196999132633209]}, {"step": 2500, "epoch": 5, "train_loss": 0.04513147100806236, "cos_r_negg": [0.29577139019966125, 0.36760804057121277, 0.3441578447818756], "cos_apical_negg": [0.06950408965349197, 0.13895860314369202, 0.2978608012199402], "cos_Ac_negg": [0.29578518867492676, 0.3675388991832733, 0.3440762162208557], "r_norm": [0.3791693150997162, 0.368236243724823, 0.36595118045806885], "ldr": {"dL_sdil": -0.009107112884521484, "dL_bp": -0.009832099080085754, "ratio": 0.9262633350560228}}, {"step": 2600, "epoch": 5, "train_loss": 0.0934981182217598, "cos_r_negg": [0.3214896321296692, 0.39022743701934814, 0.3800325393676758], "cos_apical_negg": [0.05411633104085922, 0.14016154408454895, 0.3429078757762909], "cos_Ac_negg": [0.32152223587036133, 0.390146404504776, 0.37996140122413635], "r_norm": [0.3271043300628662, 0.31700441241264343, 0.31523042917251587]}, {"step": 2700, "epoch": 5, "train_loss": 0.0971001386642456, "cos_r_negg": [0.3392976224422455, 0.38967713713645935, 0.39619553089141846], "cos_apical_negg": [0.0893879234790802, 0.1420595794916153, 0.2971969246864319], "cos_Ac_negg": [0.33929216861724854, 0.3896039128303528, 0.39616307616233826], "r_norm": [0.35043787956237793, 0.33896341919898987, 0.34025537967681885]}, {"step": 2800, "epoch": 5, "train_loss": 0.1256164312362671, "cos_r_negg": [0.3236563801765442, 0.37178584933280945, 0.42420494556427], "cos_apical_negg": [0.0778699517250061, 0.1484702229499817, 0.3644770681858063], "cos_Ac_negg": [0.32358455657958984, 0.3716598153114319, 0.4241813123226166], "r_norm": [0.3412046432495117, 0.32874637842178345, 0.32860463857650757]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9564, "test_loss": 0.1780750129699707}, {"step": 2900, "epoch": 6, "train_loss": 0.041379958391189575, "cos_r_negg": [0.27906250953674316, 0.34929847717285156, 0.4314815402030945], "cos_apical_negg": [0.0627644956111908, 0.17768119275569916, 0.3501538634300232], "cos_Ac_negg": [0.2790553569793701, 0.34919771552085876, 0.4314073622226715], "r_norm": [0.34956032037734985, 0.3371644616127014, 0.3382599353790283]}, {"step": 3000, "epoch": 6, "train_loss": 0.34214505553245544, "cos_r_negg": [0.27627718448638916, 0.3222602605819702, 0.3395204544067383], "cos_apical_negg": [0.053199075162410736, 0.09846411645412445, 0.2411688268184662], "cos_Ac_negg": [0.2762613296508789, 0.32217276096343994, 0.3394371271133423], "r_norm": [0.30777665972709656, 0.2951382100582123, 0.295731782913208], "ldr": {"dL_sdil": -0.004803195595741272, "dL_bp": -0.006155624985694885, "ratio": 0.7802937324647722}}, {"step": 3100, "epoch": 6, "train_loss": 0.1299528181552887, "cos_r_negg": [0.2844964563846588, 0.2894437909126282, 0.4082588255405426], "cos_apical_negg": [0.07756244391202927, 0.11715252697467804, 0.1790253221988678], "cos_Ac_negg": [0.28450191020965576, 0.2893946170806885, 0.40820980072021484], "r_norm": [0.35087454319000244, 0.3400832414627075, 0.3381013572216034]}, {"step": 3200, "epoch": 6, "train_loss": 0.14603891968727112, "cos_r_negg": [0.30431196093559265, 0.3414464592933655, 0.3532407879829407], "cos_apical_negg": [0.05583346262574196, 0.08824919164180756, 0.1684969961643219], "cos_Ac_negg": [0.30429357290267944, 0.34138715267181396, 0.3532252013683319], "r_norm": [0.2956363558769226, 0.28516098856925964, 0.28568029403686523]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9572, "test_loss": 0.1773396816253662}, {"step": 3300, "epoch": 7, "train_loss": 0.060805972665548325, "cos_r_negg": [0.28383898735046387, 0.3337588310241699, 0.36710286140441895], "cos_apical_negg": [0.043250832706689835, 0.1290111243724823, 0.23789039254188538], "cos_Ac_negg": [0.2838148772716522, 0.33361512422561646, 0.3670954704284668], "r_norm": [0.2922723889350891, 0.2817229926586151, 0.2819357216358185]}, {"step": 3400, "epoch": 7, "train_loss": 0.2055455446243286, "cos_r_negg": [0.27874404191970825, 0.3272626996040344, 0.3254462778568268], "cos_apical_negg": [0.08251073956489563, 0.10467558354139328, 0.20164358615875244], "cos_Ac_negg": [0.2786571681499481, 0.3271692991256714, 0.32538002729415894], "r_norm": [0.31691062450408936, 0.3045899569988251, 0.30485498905181885]}, {"step": 3500, "epoch": 7, "train_loss": 0.10245728492736816, "cos_r_negg": [0.26824331283569336, 0.32757097482681274, 0.3487533926963806], "cos_apical_negg": [0.0619991160929203, 0.09457815438508987, 0.21863090991973877], "cos_Ac_negg": [0.2682163417339325, 0.32750052213668823, 0.34871795773506165], "r_norm": [0.3403148651123047, 0.32707229256629944, 0.32713019847869873], "ldr": {"dL_sdil": -0.006241738796234131, "dL_bp": -0.0080699622631073, "ratio": 0.7734532817791367}}, {"step": 3600, "epoch": 7, "train_loss": 0.03638872131705284, "cos_r_negg": [0.27593857049942017, 0.31196749210357666, 0.32632380723953247], "cos_apical_negg": [0.032105714082717896, 0.08662648499011993, 0.23128540813922882], "cos_Ac_negg": [0.27592968940734863, 0.31189513206481934, 0.32622987031936646], "r_norm": [0.3004930019378662, 0.2896602153778076, 0.28952687978744507]}, {"step": 3700, "epoch": 7, "train_loss": 0.06841079145669937, "cos_r_negg": [0.27148646116256714, 0.33184802532196045, 0.3303089737892151], "cos_apical_negg": [0.04844781011343002, 0.10810357332229614, 0.2396182119846344], "cos_Ac_negg": [0.27147847414016724, 0.33177340030670166, 0.3302474319934845], "r_norm": [0.31027713418006897, 0.29871368408203125, 0.2979956269264221]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9594, "test_loss": 0.17211565971374512}, {"step": 3800, "epoch": 8, "train_loss": 0.055494487285614014, "cos_r_negg": [0.2379501610994339, 0.28925734758377075, 0.3242334723472595], "cos_apical_negg": [0.04948269948363304, 0.10142925381660461, 0.28163570165634155], "cos_Ac_negg": [0.23792357742786407, 0.28920406103134155, 0.32419145107269287], "r_norm": [0.2887183427810669, 0.27809247374534607, 0.2782195508480072]}, {"step": 3900, "epoch": 8, "train_loss": 0.2021137773990631, "cos_r_negg": [0.2594875991344452, 0.31504377722740173, 0.312594473361969], "cos_apical_negg": [0.026851417496800423, 0.09658454358577728, 0.21139296889305115], "cos_Ac_negg": [0.2594544291496277, 0.31502121686935425, 0.3125804662704468], "r_norm": [0.31115344166755676, 0.2986151874065399, 0.2989969253540039]}, {"step": 4000, "epoch": 8, "train_loss": 0.09608633816242218, "cos_r_negg": [0.2210184931755066, 0.26909708976745605, 0.27979251742362976], "cos_apical_negg": [0.037123903632164, 0.07577767968177795, 0.1690133959054947], "cos_Ac_negg": [0.22095856070518494, 0.2690652310848236, 0.2797449827194214], "r_norm": [0.2945510149002075, 0.2837965488433838, 0.2839779853820801], "ldr": {"dL_sdil": -0.004758194088935852, "dL_bp": -0.0057487934827804565, "ratio": 0.8276856879949196}}, {"step": 4100, "epoch": 8, "train_loss": 0.04448690265417099, "cos_r_negg": [0.24987395107746124, 0.3037770092487335, 0.3241797089576721], "cos_apical_negg": [0.03907449543476105, 0.0750594288110733, 0.2098713219165802], "cos_Ac_negg": [0.24984955787658691, 0.30371013283729553, 0.3241608738899231], "r_norm": [0.28570955991744995, 0.2731584310531616, 0.27527064085006714]}, {"step": 4200, "epoch": 8, "train_loss": 0.21926257014274597, "cos_r_negg": [0.2690197825431824, 0.3184193968772888, 0.30807825922966003], "cos_apical_negg": [0.051653966307640076, 0.09576829522848129, 0.22929464280605316], "cos_Ac_negg": [0.2690441608428955, 0.31837600469589233, 0.308036744594574], "r_norm": [0.30632734298706055, 0.295185387134552, 0.29503560066223145]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9591, "test_loss": 0.18215001335144043}, {"step": 4300, "epoch": 9, "train_loss": 0.06056637316942215, "cos_r_negg": [0.2375640720129013, 0.2759319543838501, 0.3089990019798279], "cos_apical_negg": [0.03297046944499016, 0.09315337240695953, 0.19694852828979492], "cos_Ac_negg": [0.2375570833683014, 0.2758650481700897, 0.3090273141860962], "r_norm": [0.28082606196403503, 0.26998332142829895, 0.27087530493736267]}, {"step": 4400, "epoch": 9, "train_loss": 0.057083841413259506, "cos_r_negg": [0.2541005611419678, 0.308647096157074, 0.3306451439857483], "cos_apical_negg": [0.03590516746044159, 0.07734210789203644, 0.20197394490242004], "cos_Ac_negg": [0.2541109323501587, 0.3086484670639038, 0.3306594491004944], "r_norm": [0.32942909002304077, 0.31473660469055176, 0.317359983921051]}, {"step": 4500, "epoch": 9, "train_loss": 0.17264485359191895, "cos_r_negg": [0.20887184143066406, 0.26254981756210327, 0.2650260925292969], "cos_apical_negg": [0.029389288276433945, 0.09439267218112946, 0.2195759117603302], "cos_Ac_negg": [0.20885686576366425, 0.2624436318874359, 0.2650618553161621], "r_norm": [0.3040929138660431, 0.29136353731155396, 0.2927548289299011], "ldr": {"dL_sdil": -0.005173087120056152, "dL_bp": -0.00791679322719574, "ratio": 0.6534321374323105}}, {"step": 4600, "epoch": 9, "train_loss": 0.0752175971865654, "cos_r_negg": [0.22289590537548065, 0.2698022127151489, 0.28208571672439575], "cos_apical_negg": [0.03619851917028427, 0.085477314889431, 0.24939176440238953], "cos_Ac_negg": [0.2228788584470749, 0.2697603702545166, 0.2820448577404022], "r_norm": [0.31468111276626587, 0.3012823462486267, 0.3018352687358856]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9591, "test_loss": 0.18390218124389648}, {"step": 4700, "epoch": 10, "train_loss": 0.07721929252147675, "cos_r_negg": [0.1966843456029892, 0.23706476390361786, 0.2741895318031311], "cos_apical_negg": [0.04599922522902489, 0.06404320150613785, 0.14993104338645935], "cos_Ac_negg": [0.1966865360736847, 0.23700743913650513, 0.2741366922855377], "r_norm": [0.27993515133857727, 0.26831120252609253, 0.2700636386871338]}, {"step": 4800, "epoch": 10, "train_loss": 0.04933841526508331, "cos_r_negg": [0.22418658435344696, 0.26583805680274963, 0.3130151033401489], "cos_apical_negg": [0.046518005430698395, 0.0524347648024559, 0.1983204036951065], "cos_Ac_negg": [0.22416317462921143, 0.26579636335372925, 0.3130127489566803], "r_norm": [0.2925627827644348, 0.2816404104232788, 0.28210076689720154]}, {"step": 4900, "epoch": 10, "train_loss": 0.03420368582010269, "cos_r_negg": [0.18807004392147064, 0.23366165161132812, 0.27562859654426575], "cos_apical_negg": [0.03951200097799301, 0.09651213139295578, 0.17248132824897766], "cos_Ac_negg": [0.1880582720041275, 0.23363134264945984, 0.27558690309524536], "r_norm": [0.2823236584663391, 0.27097827196121216, 0.27296847105026245]}, {"step": 5000, "epoch": 10, "train_loss": 0.04745296761393547, "cos_r_negg": [0.2138243317604065, 0.2755025625228882, 0.3482120633125305], "cos_apical_negg": [0.037016212940216064, 0.0758054107427597, 0.25266584753990173], "cos_Ac_negg": [0.21386215090751648, 0.27552640438079834, 0.3482104539871216], "r_norm": [0.3235507309436798, 0.30904749035835266, 0.3109239935874939], "ldr": {"dL_sdil": -0.0061250776052474976, "dL_bp": -0.00871327519416809, "ratio": 0.7029592740680441}}, {"step": 5100, "epoch": 10, "train_loss": 0.07249965518712997, "cos_r_negg": [0.22767964005470276, 0.26825857162475586, 0.2804119884967804], "cos_apical_negg": [0.04583592712879181, 0.08559513092041016, 0.15423330664634705], "cos_Ac_negg": [0.22763952612876892, 0.268194317817688, 0.28042399883270264], "r_norm": [0.291387677192688, 0.28049182891845703, 0.28076058626174927]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9587, "test_loss": 0.18065969924926759}, {"step": 5200, "epoch": 11, "train_loss": 0.017667211592197418, "cos_r_negg": [0.2213592827320099, 0.27206921577453613, 0.29410693049430847], "cos_apical_negg": [0.056591518223285675, 0.08806470781564713, 0.14317181706428528], "cos_Ac_negg": [0.22131603956222534, 0.2720063626766205, 0.2940799593925476], "r_norm": [0.2979279160499573, 0.2849676012992859, 0.28668737411499023]}, {"step": 5300, "epoch": 11, "train_loss": 0.07039394974708557, "cos_r_negg": [0.21036005020141602, 0.2557890713214874, 0.3030744791030884], "cos_apical_negg": [0.047831885516643524, 0.1077776849269867, 0.1418984830379486], "cos_Ac_negg": [0.21038542687892914, 0.25579971075057983, 0.303048312664032], "r_norm": [0.29446399211883545, 0.28210651874542236, 0.28310051560401917]}, {"step": 5400, "epoch": 11, "train_loss": 0.1389809548854828, "cos_r_negg": [0.18942250311374664, 0.24828064441680908, 0.2568184733390808], "cos_apical_negg": [0.017937052994966507, 0.07540638744831085, 0.16169123351573944], "cos_Ac_negg": [0.1894456148147583, 0.2482515573501587, 0.256881982088089], "r_norm": [0.26323065161705017, 0.25284603238105774, 0.25425899028778076]}, {"step": 5500, "epoch": 11, "train_loss": 0.07426370680332184, "cos_r_negg": [0.21561381220817566, 0.24225088953971863, 0.28007450699806213], "cos_apical_negg": [0.044352926313877106, 0.07818645238876343, 0.1871023327112198], "cos_Ac_negg": [0.21556456387043, 0.24218611419200897, 0.28008145093917847], "r_norm": [0.2965092360973358, 0.2840360999107361, 0.2850237786769867], "ldr": {"dL_sdil": -0.006513118743896484, "dL_bp": -0.010754704475402832, "ratio": 0.6056064821485972}}, {"step": 5600, "epoch": 11, "train_loss": 0.010568395256996155, "cos_r_negg": [0.21170857548713684, 0.24074894189834595, 0.26868629455566406], "cos_apical_negg": [0.03770392760634422, 0.0757589116692543, 0.16171687841415405], "cos_Ac_negg": [0.2116708755493164, 0.24069292843341827, 0.26865971088409424], "r_norm": [0.2969200611114502, 0.28489241003990173, 0.28569772839546204]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9601, "test_loss": 0.18408200645446776}, {"step": 5700, "epoch": 12, "train_loss": 0.05423484742641449, "cos_r_negg": [0.20610107481479645, 0.250031054019928, 0.26989874243736267], "cos_apical_negg": [0.03498787060379982, 0.08079690486192703, 0.2153046429157257], "cos_Ac_negg": [0.20606306195259094, 0.25000301003456116, 0.2700352668762207], "r_norm": [0.26945945620536804, 0.2579163908958435, 0.2595214247703552]}, {"step": 5800, "epoch": 12, "train_loss": 0.03092847391963005, "cos_r_negg": [0.20888328552246094, 0.2365681380033493, 0.29691624641418457], "cos_apical_negg": [0.04725015163421631, 0.08541767299175262, 0.17507749795913696], "cos_Ac_negg": [0.2088392674922943, 0.2365473508834839, 0.29690492153167725], "r_norm": [0.27622243762016296, 0.2644508481025696, 0.2654731273651123]}, {"step": 5900, "epoch": 12, "train_loss": 0.034352608025074005, "cos_r_negg": [0.17960354685783386, 0.2192736566066742, 0.26465660333633423], "cos_apical_negg": [0.04020436108112335, 0.06893104314804077, 0.16025002300739288], "cos_Ac_negg": [0.17960572242736816, 0.21927610039710999, 0.2646784782409668], "r_norm": [0.2822814881801605, 0.2698795795440674, 0.2713061571121216]}, {"step": 6000, "epoch": 12, "train_loss": 0.030067335814237595, "cos_r_negg": [0.19201797246932983, 0.2104051113128662, 0.30097487568855286], "cos_apical_negg": [0.055199034512043, 0.08216610550880432, 0.16139814257621765], "cos_Ac_negg": [0.1920022815465927, 0.21032708883285522, 0.30093929171562195], "r_norm": [0.31174176931381226, 0.2978854477405548, 0.2987750172615051], "ldr": {"dL_sdil": -0.00710730254650116, "dL_bp": -0.009356006979942322, "ratio": 0.7596512659447562}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9615, "test_loss": 0.18627358894348145}, {"step": 6100, "epoch": 13, "train_loss": 0.04072863608598709, "cos_r_negg": [0.17546366155147552, 0.21122168004512787, 0.24398577213287354], "cos_apical_negg": [0.0411820262670517, 0.05403941124677658, 0.17221496999263763], "cos_Ac_negg": [0.1754714548587799, 0.21115264296531677, 0.24401018023490906], "r_norm": [0.270816832780838, 0.25968310236930847, 0.26091817021369934]}, {"step": 6200, "epoch": 13, "train_loss": 0.0243786983191967, "cos_r_negg": [0.1795467734336853, 0.21211552619934082, 0.24943940341472626], "cos_apical_negg": [0.016638685017824173, 0.06957788020372391, 0.18678492307662964], "cos_Ac_negg": [0.17955666780471802, 0.2120814323425293, 0.24953234195709229], "r_norm": [0.25884437561035156, 0.2469954490661621, 0.2491001933813095]}, {"step": 6300, "epoch": 13, "train_loss": 0.08152099698781967, "cos_r_negg": [0.1668044626712799, 0.19989478588104248, 0.2628723680973053], "cos_apical_negg": [0.028372405096888542, 0.05827978998422623, 0.15671584010124207], "cos_Ac_negg": [0.16669470071792603, 0.1998569369316101, 0.2628690004348755], "r_norm": [0.27642008662223816, 0.2646355628967285, 0.26592689752578735]}, {"step": 6400, "epoch": 13, "train_loss": 0.0580926239490509, "cos_r_negg": [0.19312119483947754, 0.22229841351509094, 0.2774145305156708], "cos_apical_negg": [0.032133400440216064, 0.0813821405172348, 0.1635916829109192], "cos_Ac_negg": [0.19315162301063538, 0.22226916253566742, 0.2774478495121002], "r_norm": [0.30658918619155884, 0.2912878692150116, 0.2935417890548706]}, {"step": 6500, "epoch": 13, "train_loss": 0.005591629538685083, "cos_r_negg": [0.1965150237083435, 0.23062925040721893, 0.24952132999897003], "cos_apical_negg": [0.047904156148433685, 0.0537736751139164, 0.12270792573690414], "cos_Ac_negg": [0.19651466608047485, 0.23068967461585999, 0.2494906634092331], "r_norm": [0.2683509290218353, 0.2550716698169708, 0.2575607895851135], "ldr": {"dL_sdil": -0.005764961242675781, "dL_bp": -0.009291961789131165, "ratio": 0.6204245533401863}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9602, "test_loss": 0.1887426570892334}, {"step": 6600, "epoch": 14, "train_loss": 0.033564284443855286, "cos_r_negg": [0.15905055403709412, 0.21370354294776917, 0.24354621767997742], "cos_apical_negg": [0.037060972303152084, 0.04435315728187561, 0.1333831399679184], "cos_Ac_negg": [0.1589794009923935, 0.2136426568031311, 0.243565171957016], "r_norm": [0.2601277828216553, 0.2484085112810135, 0.24964506924152374]}, {"step": 6700, "epoch": 14, "train_loss": 0.057931434363126755, "cos_r_negg": [0.17561273276805878, 0.21444493532180786, 0.26210662722587585], "cos_apical_negg": [0.026238270103931427, 0.05927834287285805, 0.17970684170722961], "cos_Ac_negg": [0.17562194168567657, 0.2144266963005066, 0.26206517219543457], "r_norm": [0.28001832962036133, 0.26702648401260376, 0.2682105302810669]}, {"step": 6800, "epoch": 14, "train_loss": 0.08162819594144821, "cos_r_negg": [0.1556033492088318, 0.20279154181480408, 0.2468697875738144], "cos_apical_negg": [0.04083441197872162, 0.07081535458564758, 0.10019386559724808], "cos_Ac_negg": [0.15570971369743347, 0.20281541347503662, 0.2469877153635025], "r_norm": [0.275912344455719, 0.26399606466293335, 0.2650752663612366]}, {"step": 6900, "epoch": 14, "train_loss": 0.043335363268852234, "cos_r_negg": [0.15936440229415894, 0.20607733726501465, 0.23480677604675293], "cos_apical_negg": [0.025776928290724754, 0.06425251811742783, 0.13836276531219482], "cos_Ac_negg": [0.15931743383407593, 0.20606312155723572, 0.23489713668823242], "r_norm": [0.26348352432250977, 0.25113725662231445, 0.25322505831718445]}, {"step": 7000, "epoch": 14, "train_loss": 0.07403210550546646, "cos_r_negg": [0.1728387176990509, 0.22299912571907043, 0.2353971302509308], "cos_apical_negg": [0.046100541949272156, 0.07392027974128723, 0.13622358441352844], "cos_Ac_negg": [0.17284350097179413, 0.2229929268360138, 0.23542022705078125], "r_norm": [0.27658048272132874, 0.264566034078598, 0.2655790448188782], "ldr": {"dL_sdil": -0.004432976245880127, "dL_bp": -0.0077589452266693115, "ratio": 0.5713374842037742}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9587, "test_loss": 0.1950464973449707}], "final": {"test_acc": 0.9587, "test_loss": 0.1950464973449707, "wall_s": 43.170592069625854, "cos_r_negg": [0.15515871345996857, 0.19200049340724945, 0.23033785820007324], "cos_apical_negg": [0.04083329066634178, 0.06870468705892563, 0.09320558607578278], "cos_Ac_negg": [0.15518587827682495, 0.19202908873558044, 0.23038293421268463]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p1_res1_s0.json b/results/diag_nuis_low_rho0p1_res1_s0.json new file mode 100644 index 0000000..904593f --- /dev/null +++ b/results/diag_nuis_low_rho0p1_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.1, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p1_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.019516456872224808, -0.025525914505124092, 0.033765003085136414], "cos_apical_negg": [-0.019420741125941277, -0.025776736438274384, 0.032360587269067764], "cos_Ac_negg": [-0.019516363739967346, -0.025525912642478943, 0.03376498818397522], "r_norm": [4.898177146911621, 4.826803684234619, 4.823856830596924], "ldr": {"dL_sdil": -0.009199380874633789, "dL_bp": -0.24068236351013184, "ratio": 0.038222081337698555}}, {"step": 100, "epoch": 0, "train_loss": 0.20683997869491577, "cos_r_negg": [0.4561299681663513, 0.3808678686618805, 0.36012405157089233], "cos_apical_negg": [0.15237829089164734, 0.143134206533432, 0.18917536735534668], "cos_Ac_negg": [0.4566975235939026, 0.3814869523048401, 0.3605459928512573], "r_norm": [0.8242521286010742, 0.8076595664024353, 0.8054618835449219]}, {"step": 200, "epoch": 0, "train_loss": 0.32852238416671753, "cos_r_negg": [0.4740169048309326, 0.4043392837047577, 0.3805192708969116], "cos_apical_negg": [0.13903754949569702, 0.14106644690036774, 0.18174822628498077], "cos_Ac_negg": [0.4749114513397217, 0.4056163430213928, 0.382116436958313], "r_norm": [0.7472480535507202, 0.7297910451889038, 0.7290124893188477]}, {"step": 300, "epoch": 0, "train_loss": 0.1792018860578537, "cos_r_negg": [0.5064927935600281, 0.46594464778900146, 0.4275602102279663], "cos_apical_negg": [0.1177990511059761, 0.13220566511154175, 0.16945992410182953], "cos_Ac_negg": [0.5072965621948242, 0.46631813049316406, 0.4278665781021118], "r_norm": [0.5790054798126221, 0.5678757429122925, 0.5665429830551147]}, {"step": 400, "epoch": 0, "train_loss": 0.16270679235458374, "cos_r_negg": [0.5334340333938599, 0.4902327358722687, 0.4517212510108948], "cos_apical_negg": [0.12907904386520386, 0.12549903988838196, 0.17313998937606812], "cos_Ac_negg": [0.5339264869689941, 0.49035918712615967, 0.45164915919303894], "r_norm": [0.5338784456253052, 0.5195522904396057, 0.5205836296081543]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9454, "test_loss": 0.1904225929260254}, {"step": 500, "epoch": 1, "train_loss": 0.23014771938323975, "cos_r_negg": [0.5086731910705566, 0.4808811545372009, 0.47930672764778137], "cos_apical_negg": [0.11290472000837326, 0.11339855194091797, 0.1659645140171051], "cos_Ac_negg": [0.5092896223068237, 0.48063361644744873, 0.47944051027297974], "r_norm": [0.5010726451873779, 0.48585939407348633, 0.4869433045387268], "ldr": {"dL_sdil": -0.012213855981826782, "dL_bp": -0.010371357202529907, "ratio": 1.1776526199336217}}, {"step": 600, "epoch": 1, "train_loss": 0.06264102458953857, "cos_r_negg": [0.5161782503128052, 0.49555227160453796, 0.47790834307670593], "cos_apical_negg": [0.1079772338271141, 0.10634440183639526, 0.16544830799102783], "cos_Ac_negg": [0.5170495510101318, 0.4960915446281433, 0.47849780321121216], "r_norm": [0.4478020668029785, 0.4375015199184418, 0.4376702308654785]}, {"step": 700, "epoch": 1, "train_loss": 0.21048517525196075, "cos_r_negg": [0.5166497826576233, 0.5061420202255249, 0.48557013273239136], "cos_apical_negg": [0.11987586319446564, 0.11258520931005478, 0.17083010077476501], "cos_Ac_negg": [0.518173336982727, 0.5066292881965637, 0.4856169819831848], "r_norm": [0.48917144536972046, 0.47349128127098083, 0.4745555520057678]}, {"step": 800, "epoch": 1, "train_loss": 0.1698085069656372, "cos_r_negg": [0.5281230211257935, 0.5412549376487732, 0.49062949419021606], "cos_apical_negg": [0.08528990298509598, 0.10898679494857788, 0.1691044569015503], "cos_Ac_negg": [0.5291732549667358, 0.5413188934326172, 0.4903642535209656], "r_norm": [0.37939468026161194, 0.36847028136253357, 0.36765846610069275]}, {"step": 900, "epoch": 1, "train_loss": 0.17834270000457764, "cos_r_negg": [0.5426621437072754, 0.5571110844612122, 0.46928098797798157], "cos_apical_negg": [0.08367139846086502, 0.1062052771449089, 0.1484588384628296], "cos_Ac_negg": [0.5428537726402283, 0.5568511486053467, 0.46884000301361084], "r_norm": [0.3773725628852844, 0.36773693561553955, 0.3660231828689575]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9532, "test_loss": 0.15338984069824219}, {"step": 1000, "epoch": 2, "train_loss": 0.03610467538237572, "cos_r_negg": [0.5278955698013306, 0.5470715165138245, 0.4887616038322449], "cos_apical_negg": [0.08191273361444473, 0.0979018360376358, 0.1530991494655609], "cos_Ac_negg": [0.528154730796814, 0.5465915203094482, 0.48795825242996216], "r_norm": [0.3610769808292389, 0.34987568855285645, 0.3488883674144745], "ldr": {"dL_sdil": -0.00946018099784851, "dL_bp": -0.008130252361297607, "ratio": 1.1635777805473486}}, {"step": 1100, "epoch": 2, "train_loss": 0.14434075355529785, "cos_r_negg": [0.5304457545280457, 0.547247588634491, 0.4867292046546936], "cos_apical_negg": [0.07937635481357574, 0.09804007411003113, 0.15954582393169403], "cos_Ac_negg": [0.5308558940887451, 0.546707034111023, 0.48520904779434204], "r_norm": [0.3578449487686157, 0.34673258662223816, 0.34743696451187134]}, {"step": 1200, "epoch": 2, "train_loss": 0.1277787983417511, "cos_r_negg": [0.5395125150680542, 0.558314859867096, 0.5030388832092285], "cos_apical_negg": [0.07928188145160675, 0.09109491109848022, 0.15419000387191772], "cos_Ac_negg": [0.5396093130111694, 0.5580030679702759, 0.5018350481987], "r_norm": [0.3390808701515198, 0.33018729090690613, 0.3292943835258484]}, {"step": 1300, "epoch": 2, "train_loss": 0.11057958006858826, "cos_r_negg": [0.515335202217102, 0.5336487293243408, 0.4851382374763489], "cos_apical_negg": [0.06864278763532639, 0.09958652406930923, 0.15436717867851257], "cos_Ac_negg": [0.5155478119850159, 0.5326103568077087, 0.4835350513458252], "r_norm": [0.34386441111564636, 0.33492058515548706, 0.3315686583518982]}, {"step": 1400, "epoch": 2, "train_loss": 0.1388072520494461, "cos_r_negg": [0.5155563354492188, 0.5309954285621643, 0.4800029397010803], "cos_apical_negg": [0.07199728488922119, 0.09795089811086655, 0.15476271510124207], "cos_Ac_negg": [0.5155479907989502, 0.5297256708145142, 0.47807013988494873], "r_norm": [0.27236801385879517, 0.26437029242515564, 0.2636326849460602]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9649, "test_loss": 0.12148791046142578}, {"step": 1500, "epoch": 3, "train_loss": 0.0676574856042862, "cos_r_negg": [0.4874963164329529, 0.5175042152404785, 0.4765663146972656], "cos_apical_negg": [0.06620067358016968, 0.08575887978076935, 0.153273344039917], "cos_Ac_negg": [0.48696964979171753, 0.5163366794586182, 0.47409555315971375], "r_norm": [0.2755729556083679, 0.2680559754371643, 0.26864421367645264], "ldr": {"dL_sdil": -0.0059937983751297, "dL_bp": -0.006250232458114624, "ratio": 0.958972072686353}}, {"step": 1600, "epoch": 3, "train_loss": 0.07317554950714111, "cos_r_negg": [0.5015812516212463, 0.5151363611221313, 0.4900739789009094], "cos_apical_negg": [0.05664137378334999, 0.08271725475788116, 0.15622611343860626], "cos_Ac_negg": [0.5016165971755981, 0.514200747013092, 0.48732006549835205], "r_norm": [0.2753051519393921, 0.2664317786693573, 0.2673736810684204]}, {"step": 1700, "epoch": 3, "train_loss": 0.08164312690496445, "cos_r_negg": [0.4873105585575104, 0.5137908458709717, 0.4737226665019989], "cos_apical_negg": [0.06819788366556168, 0.08815443515777588, 0.14751389622688293], "cos_Ac_negg": [0.4867975115776062, 0.5132312178611755, 0.47131606936454773], "r_norm": [0.24582251906394958, 0.23820431530475616, 0.23781003057956696]}, {"step": 1800, "epoch": 3, "train_loss": 0.024689778685569763, "cos_r_negg": [0.501431941986084, 0.5187755823135376, 0.48886168003082275], "cos_apical_negg": [0.05063096061348915, 0.07646887004375458, 0.1549108773469925], "cos_Ac_negg": [0.5011186599731445, 0.5184181928634644, 0.4863644540309906], "r_norm": [0.2637336552143097, 0.25648945569992065, 0.2538714110851288]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.965, "test_loss": 0.12390026512145996}, {"step": 1900, "epoch": 4, "train_loss": 0.03273295983672142, "cos_r_negg": [0.5006571412086487, 0.5333853960037231, 0.4936748445034027], "cos_apical_negg": [0.049160655587911606, 0.09003491699695587, 0.15443134307861328], "cos_Ac_negg": [0.5006347894668579, 0.53319251537323, 0.49140793085098267], "r_norm": [0.2740962505340576, 0.2679555416107178, 0.265875905752182]}, {"step": 2000, "epoch": 4, "train_loss": 0.09164918959140778, "cos_r_negg": [0.4288943409919739, 0.4692956805229187, 0.440817654132843], "cos_apical_negg": [0.036496445536613464, 0.07540971785783768, 0.13341930508613586], "cos_Ac_negg": [0.4288756847381592, 0.4682854115962982, 0.43775996565818787], "r_norm": [0.21223405003547668, 0.20557604730129242, 0.20465439558029175], "ldr": {"dL_sdil": -0.004755005240440369, "dL_bp": -0.005659237504005432, "ratio": 0.84022012454415}}, {"step": 2100, "epoch": 4, "train_loss": 0.01425383985042572, "cos_r_negg": [0.44933074712753296, 0.4964136481285095, 0.47037753462791443], "cos_apical_negg": [0.03857060894370079, 0.0800505131483078, 0.1505504846572876], "cos_Ac_negg": [0.44901490211486816, 0.4954824447631836, 0.46752554178237915], "r_norm": [0.24937845766544342, 0.24185745418071747, 0.2415986955165863]}, {"step": 2200, "epoch": 4, "train_loss": 0.010673163458704948, "cos_r_negg": [0.4274923503398895, 0.4522479176521301, 0.4385167360305786], "cos_apical_negg": [0.04543246328830719, 0.07618136703968048, 0.15272898972034454], "cos_Ac_negg": [0.42780643701553345, 0.4516759514808655, 0.4355747103691101], "r_norm": [0.2627090811729431, 0.25427308678627014, 0.25260934233665466]}, {"step": 2300, "epoch": 4, "train_loss": 0.1186101883649826, "cos_r_negg": [0.4036998748779297, 0.43460994958877563, 0.41644877195358276], "cos_apical_negg": [0.050553448498249054, 0.07390330731868744, 0.1347779929637909], "cos_Ac_negg": [0.40327024459838867, 0.4336010217666626, 0.4131975769996643], "r_norm": [0.222402885556221, 0.21528461575508118, 0.21474608778953552]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9686, "test_loss": 0.12125136375427246}, {"step": 2400, "epoch": 5, "train_loss": 0.0777900218963623, "cos_r_negg": [0.4097362160682678, 0.45188161730766296, 0.4314563274383545], "cos_apical_negg": [0.0326840840280056, 0.06365515291690826, 0.14021185040473938], "cos_Ac_negg": [0.40931063890457153, 0.450900137424469, 0.42771705985069275], "r_norm": [0.21444934606552124, 0.2079712450504303, 0.20672295987606049]}, {"step": 2500, "epoch": 5, "train_loss": 0.06600132584571838, "cos_r_negg": [0.3701138198375702, 0.4027906656265259, 0.38374143838882446], "cos_apical_negg": [0.03292577341198921, 0.07368376106023788, 0.1426790952682495], "cos_Ac_negg": [0.3699338138103485, 0.40199804306030273, 0.3807746469974518], "r_norm": [0.20255105197429657, 0.1969395875930786, 0.19603750109672546], "ldr": {"dL_sdil": -0.005280748009681702, "dL_bp": -0.005878016352653503, "ratio": 0.89838947237665}}, {"step": 2600, "epoch": 5, "train_loss": 0.030449341982603073, "cos_r_negg": [0.3600800633430481, 0.3967919945716858, 0.37056976556777954], "cos_apical_negg": [0.03236114978790283, 0.061309609562158585, 0.14484903216362], "cos_Ac_negg": [0.35986825823783875, 0.39589923620224, 0.36683547496795654], "r_norm": [0.1689160168170929, 0.1657443642616272, 0.16257992386817932]}, {"step": 2700, "epoch": 5, "train_loss": 0.0222881231456995, "cos_r_negg": [0.39370667934417725, 0.4418136477470398, 0.4159846603870392], "cos_apical_negg": [0.032427363097667694, 0.06435207277536392, 0.14682599902153015], "cos_Ac_negg": [0.39332208037376404, 0.440535306930542, 0.4115748405456543], "r_norm": [0.19391296803951263, 0.18947941064834595, 0.18828833103179932]}, {"step": 2800, "epoch": 5, "train_loss": 0.08499300479888916, "cos_r_negg": [0.37556856870651245, 0.4174972176551819, 0.39975878596305847], "cos_apical_negg": [0.03558485209941864, 0.067987360060215, 0.14555107057094574], "cos_Ac_negg": [0.3755994439125061, 0.4164520502090454, 0.3959047794342041], "r_norm": [0.2091490626335144, 0.20293818414211273, 0.20068201422691345]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9701, "test_loss": 0.11767922821044922}, {"step": 2900, "epoch": 6, "train_loss": 0.009337813593447208, "cos_r_negg": [0.35428890585899353, 0.39382070302963257, 0.3790283203125], "cos_apical_negg": [0.02073688805103302, 0.06096653267741203, 0.14554941654205322], "cos_Ac_negg": [0.35517823696136475, 0.39343321323394775, 0.3752865791320801], "r_norm": [0.20045910775661469, 0.19383198022842407, 0.1936217099428177]}, {"step": 3000, "epoch": 6, "train_loss": 0.06943769752979279, "cos_r_negg": [0.3427330255508423, 0.3803616166114807, 0.3641650676727295], "cos_apical_negg": [0.019505774602293968, 0.05789085477590561, 0.14434850215911865], "cos_Ac_negg": [0.3430076241493225, 0.37962138652801514, 0.3600381910800934], "r_norm": [0.16525718569755554, 0.16065149009227753, 0.1590198576450348], "ldr": {"dL_sdil": -0.004927143454551697, "dL_bp": -0.006980516016483307, "ratio": 0.7058422963169315}}, {"step": 3100, "epoch": 6, "train_loss": 0.011620339937508106, "cos_r_negg": [0.3101832866668701, 0.34814655780792236, 0.3256959915161133], "cos_apical_negg": [0.009094222448766232, 0.06744265556335449, 0.14300718903541565], "cos_Ac_negg": [0.31060564517974854, 0.34736382961273193, 0.32170993089675903], "r_norm": [0.18792138993740082, 0.18322056531906128, 0.1812620759010315]}, {"step": 3200, "epoch": 6, "train_loss": 0.01361056324094534, "cos_r_negg": [0.3047287166118622, 0.3504140079021454, 0.33207985758781433], "cos_apical_negg": [0.019136711955070496, 0.06709253042936325, 0.13717174530029297], "cos_Ac_negg": [0.3048957884311676, 0.34936195611953735, 0.3283121585845947], "r_norm": [0.17606475949287415, 0.1713852733373642, 0.17045295238494873]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9711, "test_loss": 0.12406212673187256}, {"step": 3300, "epoch": 7, "train_loss": 0.011202439665794373, "cos_r_negg": [0.2870562970638275, 0.3235819935798645, 0.30494198203086853], "cos_apical_negg": [0.01887010596692562, 0.05337080359458923, 0.1363985538482666], "cos_Ac_negg": [0.2873172163963318, 0.3229443430900574, 0.3012893497943878], "r_norm": [0.14607161283493042, 0.14222578704357147, 0.14072765409946442]}, {"step": 3400, "epoch": 7, "train_loss": 0.02864265814423561, "cos_r_negg": [0.2825903594493866, 0.3174539804458618, 0.3021314740180969], "cos_apical_negg": [0.019116273149847984, 0.06002703681588173, 0.14857259392738342], "cos_Ac_negg": [0.2829700708389282, 0.31706857681274414, 0.2986683249473572], "r_norm": [0.18481570482254028, 0.18003259599208832, 0.1788252890110016]}, {"step": 3500, "epoch": 7, "train_loss": 0.00422491692006588, "cos_r_negg": [0.2678760588169098, 0.3047921359539032, 0.2917213439941406], "cos_apical_negg": [0.02134549617767334, 0.05760381370782852, 0.14684244990348816], "cos_Ac_negg": [0.2683059871196747, 0.3039183020591736, 0.28749969601631165], "r_norm": [0.19979383051395416, 0.19399701058864594, 0.19357004761695862], "ldr": {"dL_sdil": -0.006273619830608368, "dL_bp": -0.009467460215091705, "ratio": 0.6626507730772228}}, {"step": 3600, "epoch": 7, "train_loss": 0.004532014485448599, "cos_r_negg": [0.26364099979400635, 0.30037909746170044, 0.2863549590110779], "cos_apical_negg": [0.0057756174355745316, 0.05669254809617996, 0.13763698935508728], "cos_Ac_negg": [0.2636473774909973, 0.29982566833496094, 0.2833068072795868], "r_norm": [0.1825418770313263, 0.17865371704101562, 0.17635765671730042]}, {"step": 3700, "epoch": 7, "train_loss": 0.006920909509062767, "cos_r_negg": [0.2698700726032257, 0.3056122958660126, 0.2946384847164154], "cos_apical_negg": [0.010869037359952927, 0.053051311522722244, 0.1392381191253662], "cos_Ac_negg": [0.2702271342277527, 0.3051472306251526, 0.29140734672546387], "r_norm": [0.19647647440433502, 0.1898655742406845, 0.1878773421049118]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9722, "test_loss": 0.11938170566558838}, {"step": 3800, "epoch": 8, "train_loss": 0.007145673036575317, "cos_r_negg": [0.23417602479457855, 0.2666628658771515, 0.2651404142379761], "cos_apical_negg": [0.0029220422729849815, 0.053463634103536606, 0.14256519079208374], "cos_Ac_negg": [0.23486799001693726, 0.2663212716579437, 0.26189178228378296], "r_norm": [0.18395251035690308, 0.17998728156089783, 0.1775822937488556]}, {"step": 3900, "epoch": 8, "train_loss": 0.0042358096688985825, "cos_r_negg": [0.21914923191070557, 0.24916185438632965, 0.24405601620674133], "cos_apical_negg": [0.0032741674222052097, 0.05763109773397446, 0.13305151462554932], "cos_Ac_negg": [0.22004622220993042, 0.24933794140815735, 0.24191449582576752], "r_norm": [0.17436614632606506, 0.16928914189338684, 0.16762040555477142]}, {"step": 4000, "epoch": 8, "train_loss": 0.0032062570098787546, "cos_r_negg": [0.21606554090976715, 0.24830974638462067, 0.24428966641426086], "cos_apical_negg": [0.00016378844156861305, 0.05589856952428818, 0.13327142596244812], "cos_Ac_negg": [0.2169175148010254, 0.24785946309566498, 0.2417692244052887], "r_norm": [0.1559789478778839, 0.15242508053779602, 0.15043389797210693], "ldr": {"dL_sdil": -0.0062768831849098206, "dL_bp": -0.0106586292386055, "ratio": 0.5889015411264126}}, {"step": 4100, "epoch": 8, "train_loss": 0.002575080143287778, "cos_r_negg": [0.2256929874420166, 0.2585234045982361, 0.2557886242866516], "cos_apical_negg": [0.011652304790914059, 0.05617436766624451, 0.13030089437961578], "cos_Ac_negg": [0.22626455128192902, 0.25815287232398987, 0.2533252239227295], "r_norm": [0.16336297988891602, 0.158944234251976, 0.1570030152797699]}, {"step": 4200, "epoch": 8, "train_loss": 0.01088833250105381, "cos_r_negg": [0.2047746777534485, 0.23343707621097565, 0.22910253703594208], "cos_apical_negg": [0.007006949745118618, 0.05109415203332901, 0.11974142491817474], "cos_Ac_negg": [0.20516666769981384, 0.23341678082942963, 0.22676023840904236], "r_norm": [0.17266660928726196, 0.1680162400007248, 0.1657750904560089]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.972, "test_loss": 0.12974063901901245}, {"step": 4300, "epoch": 9, "train_loss": 0.0030862889252603054, "cos_r_negg": [0.211567223072052, 0.2391878217458725, 0.23505696654319763], "cos_apical_negg": [0.0057335300371050835, 0.05104059725999832, 0.12385048717260361], "cos_Ac_negg": [0.21240133047103882, 0.23901788890361786, 0.23264044523239136], "r_norm": [0.1723661720752716, 0.1682838797569275, 0.16532552242279053]}, {"step": 4400, "epoch": 9, "train_loss": 0.0009592562564648688, "cos_r_negg": [0.19718368351459503, 0.22516745328903198, 0.22213834524154663], "cos_apical_negg": [0.0049959090538322926, 0.05399724468588829, 0.12144915759563446], "cos_Ac_negg": [0.19782474637031555, 0.22509709000587463, 0.21999472379684448], "r_norm": [0.16731835901737213, 0.1636153608560562, 0.16041779518127441]}, {"step": 4500, "epoch": 9, "train_loss": 0.03124357759952545, "cos_r_negg": [0.19361761212348938, 0.2217017263174057, 0.21697364747524261], "cos_apical_negg": [0.007766964845359325, 0.05152570828795433, 0.11414465308189392], "cos_Ac_negg": [0.19425377249717712, 0.22116243839263916, 0.2144576758146286], "r_norm": [0.15997500717639923, 0.15626540780067444, 0.15357846021652222], "ldr": {"dL_sdil": -0.006380163133144379, "dL_bp": -0.011209465563297272, "ratio": 0.5691763891076757}}, {"step": 4600, "epoch": 9, "train_loss": 0.0002524250012356788, "cos_r_negg": [0.1992313265800476, 0.22967037558555603, 0.22606799006462097], "cos_apical_negg": [0.008906925097107887, 0.05459585040807724, 0.1169220432639122], "cos_Ac_negg": [0.19988197088241577, 0.22920489311218262, 0.22359073162078857], "r_norm": [0.17042243480682373, 0.16543200612068176, 0.16314949095249176]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9724, "test_loss": 0.13544013366699217}, {"step": 4700, "epoch": 10, "train_loss": 0.002242384012788534, "cos_r_negg": [0.18099886178970337, 0.20816196501255035, 0.20495903491973877], "cos_apical_negg": [0.005294828675687313, 0.05115308612585068, 0.10896711051464081], "cos_Ac_negg": [0.18201428651809692, 0.20794838666915894, 0.20277822017669678], "r_norm": [0.1624893844127655, 0.15819205343723297, 0.15589958429336548]}, {"step": 4800, "epoch": 10, "train_loss": 0.0022209174931049347, "cos_r_negg": [0.17720907926559448, 0.20473989844322205, 0.2030676305294037], "cos_apical_negg": [0.007636585272848606, 0.05117449164390564, 0.10893990099430084], "cos_Ac_negg": [0.17830388247966766, 0.2046879231929779, 0.20094652473926544], "r_norm": [0.16868388652801514, 0.16412308812141418, 0.16156303882598877]}, {"step": 4900, "epoch": 10, "train_loss": 0.0020407531410455704, "cos_r_negg": [0.1728314459323883, 0.20039692521095276, 0.20057938992977142], "cos_apical_negg": [0.004512418992817402, 0.05042345076799393, 0.10664612054824829], "cos_Ac_negg": [0.17398978769779205, 0.20064817368984222, 0.19872286915779114], "r_norm": [0.16799789667129517, 0.16384561359882355, 0.16117820143699646]}, {"step": 5000, "epoch": 10, "train_loss": 0.0003883102326653898, "cos_r_negg": [0.17160284519195557, 0.19773593544960022, 0.19740310311317444], "cos_apical_negg": [-0.0014983888249844313, 0.04848548769950867, 0.10114528983831406], "cos_Ac_negg": [0.17249926924705505, 0.19801467657089233, 0.19585081934928894], "r_norm": [0.1681741178035736, 0.1642361879348755, 0.16123953461647034], "ldr": {"dL_sdil": -0.00737868994474411, "dL_bp": -0.015240125358104706, "ratio": 0.48416202435107397}}, {"step": 5100, "epoch": 10, "train_loss": 0.003490174887701869, "cos_r_negg": [0.18060022592544556, 0.20882394909858704, 0.20371171832084656], "cos_apical_negg": [0.002307755872607231, 0.04839232563972473, 0.10792817175388336], "cos_Ac_negg": [0.18092948198318481, 0.20830607414245605, 0.2014542520046234], "r_norm": [0.17668594419956207, 0.17332932353019714, 0.1698925644159317]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9712, "test_loss": 0.14135054550170897}, {"step": 5200, "epoch": 11, "train_loss": 0.0009749945602379739, "cos_r_negg": [0.1623513251543045, 0.18656250834465027, 0.18713593482971191], "cos_apical_negg": [0.00042043160647153854, 0.04728969559073448, 0.10461169481277466], "cos_Ac_negg": [0.16323581337928772, 0.18654069304466248, 0.18540030717849731], "r_norm": [0.1595700979232788, 0.15614834427833557, 0.15367180109024048]}, {"step": 5300, "epoch": 11, "train_loss": 0.004578763153403997, "cos_r_negg": [0.16511301696300507, 0.19113531708717346, 0.19296324253082275], "cos_apical_negg": [0.0067689428105950356, 0.049471739679574966, 0.09949158132076263], "cos_Ac_negg": [0.16606497764587402, 0.19137325882911682, 0.19141457974910736], "r_norm": [0.16025295853614807, 0.1566634476184845, 0.15421411395072937]}, {"step": 5400, "epoch": 11, "train_loss": 0.0030923662707209587, "cos_r_negg": [0.16054025292396545, 0.1847323775291443, 0.18501058220863342], "cos_apical_negg": [-0.0002478715032339096, 0.045806627720594406, 0.1013571247458458], "cos_Ac_negg": [0.16138291358947754, 0.18480870127677917, 0.18342041969299316], "r_norm": [0.15932536125183105, 0.15558157861232758, 0.1530345231294632]}, {"step": 5500, "epoch": 11, "train_loss": 0.0015335017815232277, "cos_r_negg": [0.15501058101654053, 0.1787438690662384, 0.17968013882637024], "cos_apical_negg": [0.002538089407607913, 0.05129781365394592, 0.10300736874341965], "cos_Ac_negg": [0.1559019386768341, 0.17877711355686188, 0.17818503081798553], "r_norm": [0.1596858948469162, 0.15611006319522858, 0.15354576706886292], "ldr": {"dL_sdil": -0.006922267377376556, "dL_bp": -0.014339148998260498, "ratio": 0.4827530126241317}}, {"step": 5600, "epoch": 11, "train_loss": 0.0027611474506556988, "cos_r_negg": [0.15004193782806396, 0.1720571666955948, 0.17380842566490173], "cos_apical_negg": [0.0020210598595440388, 0.04668239504098892, 0.09548679739236832], "cos_Ac_negg": [0.15076035261154175, 0.17231373488903046, 0.172477588057518], "r_norm": [0.15836629271507263, 0.15478134155273438, 0.1522841900587082]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9725, "test_loss": 0.14267677993774414}, {"step": 5700, "epoch": 12, "train_loss": 0.0018286288250237703, "cos_r_negg": [0.14924080669879913, 0.17178520560264587, 0.1737925261259079], "cos_apical_negg": [0.0009777843952178955, 0.045955948531627655, 0.09453579783439636], "cos_Ac_negg": [0.1501063108444214, 0.172155499458313, 0.1724928915500641], "r_norm": [0.15410643815994263, 0.15070940554141998, 0.14827893674373627]}, {"step": 5800, "epoch": 12, "train_loss": 0.00040417310083284974, "cos_r_negg": [0.14772182703018188, 0.16990524530410767, 0.17158560454845428], "cos_apical_negg": [-3.583705984055996e-05, 0.0466625839471817, 0.09365487843751907], "cos_Ac_negg": [0.148675799369812, 0.17023327946662903, 0.17027974128723145], "r_norm": [0.15465311706066132, 0.15168555080890656, 0.14889883995056152]}, {"step": 5900, "epoch": 12, "train_loss": 0.0008682627230882645, "cos_r_negg": [0.14963379502296448, 0.17269492149353027, 0.1749807596206665], "cos_apical_negg": [0.0014506014995276928, 0.045361898839473724, 0.09281644225120544], "cos_Ac_negg": [0.15051476657390594, 0.17299848794937134, 0.17364919185638428], "r_norm": [0.15576529502868652, 0.15237319469451904, 0.14976321160793304]}, {"step": 6000, "epoch": 12, "train_loss": 0.0006659608916379511, "cos_r_negg": [0.14338232576847076, 0.16516324877738953, 0.16691350936889648], "cos_apical_negg": [0.002782408380880952, 0.04511334002017975, 0.09082347899675369], "cos_Ac_negg": [0.1443859338760376, 0.16549432277679443, 0.16567006707191467], "r_norm": [0.15318021178245544, 0.1500137746334076, 0.14754506945610046], "ldr": {"dL_sdil": -0.00698409229516983, "dL_bp": -0.014542214572429657, "ratio": 0.4802633230574699}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9729, "test_loss": 0.14518474140167237}, {"step": 6100, "epoch": 13, "train_loss": 0.004160470329225063, "cos_r_negg": [0.13942095637321472, 0.16058944165706635, 0.16170142590999603], "cos_apical_negg": [0.0035982229746878147, 0.042050011456012726, 0.08914529532194138], "cos_Ac_negg": [0.14031493663787842, 0.16081255674362183, 0.1604430079460144], "r_norm": [0.15666279196739197, 0.15299075841903687, 0.1504707634449005]}, {"step": 6200, "epoch": 13, "train_loss": 0.0004260868299752474, "cos_r_negg": [0.1381671130657196, 0.15893764793872833, 0.1603722870349884], "cos_apical_negg": [0.0023114874493330717, 0.04371057450771332, 0.08661684393882751], "cos_Ac_negg": [0.13917770981788635, 0.15928177535533905, 0.1592041552066803], "r_norm": [0.15024037659168243, 0.1471148580312729, 0.14463046193122864]}, {"step": 6300, "epoch": 13, "train_loss": 0.0009916559793055058, "cos_r_negg": [0.13507600128650665, 0.1552838385105133, 0.1563580334186554], "cos_apical_negg": [0.0012639507185667753, 0.04138736426830292, 0.08471791446208954], "cos_Ac_negg": [0.13610166311264038, 0.15565194189548492, 0.15516991913318634], "r_norm": [0.15482991933822632, 0.15149924159049988, 0.14899252355098724]}, {"step": 6400, "epoch": 13, "train_loss": 0.0005770027637481689, "cos_r_negg": [0.13531386852264404, 0.15646056830883026, 0.1574159860610962], "cos_apical_negg": [0.0017425506375730038, 0.042211584746837616, 0.0863981544971466], "cos_Ac_negg": [0.13638393580913544, 0.15669916570186615, 0.15620100498199463], "r_norm": [0.15497565269470215, 0.15167251229286194, 0.14908693730831146]}, {"step": 6500, "epoch": 13, "train_loss": 0.00028461741749197245, "cos_r_negg": [0.13315211236476898, 0.15384900569915771, 0.15536534786224365], "cos_apical_negg": [0.0021088896319270134, 0.04172522574663162, 0.08452653884887695], "cos_Ac_negg": [0.13412320613861084, 0.1540856957435608, 0.1542053073644638], "r_norm": [0.15321414172649384, 0.14976288378238678, 0.14725030958652496], "ldr": {"dL_sdil": -0.007184863090515137, "dL_bp": -0.015180259943008423, "ratio": 0.473303034169996}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9731, "test_loss": 0.14860017442703247}, {"step": 6600, "epoch": 14, "train_loss": 0.0005150830838829279, "cos_r_negg": [0.13424663245677948, 0.1553696095943451, 0.15640458464622498], "cos_apical_negg": [0.003459415165707469, 0.04157208651304245, 0.08230052143335342], "cos_Ac_negg": [0.1352035105228424, 0.15559476613998413, 0.1552029848098755], "r_norm": [0.15187077224254608, 0.14889103174209595, 0.14622268080711365]}, {"step": 6700, "epoch": 14, "train_loss": 0.00037772199721075594, "cos_r_negg": [0.13177327811717987, 0.1531122326850891, 0.15459823608398438], "cos_apical_negg": [0.004713563248515129, 0.04190857708454132, 0.08322569727897644], "cos_Ac_negg": [0.13269993662834167, 0.15331876277923584, 0.15342417359352112], "r_norm": [0.15484532713890076, 0.15157616138458252, 0.14897796511650085]}, {"step": 6800, "epoch": 14, "train_loss": 0.0005252625560387969, "cos_r_negg": [0.129064679145813, 0.1494249403476715, 0.15062189102172852], "cos_apical_negg": [0.0026286919601261616, 0.039924778044223785, 0.07876899838447571], "cos_Ac_negg": [0.1300213485956192, 0.1496838629245758, 0.14954346418380737], "r_norm": [0.15402114391326904, 0.15076418220996857, 0.14815939962863922]}, {"step": 6900, "epoch": 14, "train_loss": 0.0009440174326300621, "cos_r_negg": [0.127854585647583, 0.14814069867134094, 0.1484384387731552], "cos_apical_negg": [0.0012316768988966942, 0.04094769433140755, 0.07605629414319992], "cos_Ac_negg": [0.1288197636604309, 0.14839869737625122, 0.14735260605812073], "r_norm": [0.14917528629302979, 0.14638496935367584, 0.14371880888938904]}, {"step": 7000, "epoch": 14, "train_loss": 0.0017105384031310678, "cos_r_negg": [0.1272173821926117, 0.14776089787483215, 0.1483118087053299], "cos_apical_negg": [0.002162056742236018, 0.041705384850502014, 0.07823067903518677], "cos_Ac_negg": [0.1283431351184845, 0.14804384112358093, 0.14730221033096313], "r_norm": [0.15299172699451447, 0.1500447690486908, 0.14738385379314423], "ldr": {"dL_sdil": -0.007436320185661316, "dL_bp": -0.01620202511548996, "ratio": 0.45897473511208275}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9732, "test_loss": 0.15152641515731813}], "final": {"test_acc": 0.9732, "test_loss": 0.15152641515731813, "wall_s": 39.837257385253906, "cos_r_negg": [0.12629830837249756, 0.14677634835243225, 0.14761337637901306], "cos_apical_negg": [0.003244647290557623, 0.04177744314074516, 0.07807733863592148], "cos_Ac_negg": [0.12736676633358002, 0.14705677330493927, 0.14659380912780762]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p2_res0_s0.json b/results/diag_nuis_low_rho0p2_res0_s0.json new file mode 100644 index 0000000..7aa42f9 --- /dev/null +++ b/results/diag_nuis_low_rho0p2_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.2, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p2_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.01658112369477749, -0.02675408124923706, 0.03309158980846405], "cos_apical_negg": [-0.01647482067346573, -0.02632959932088852, 0.029919130727648735], "cos_Ac_negg": [-0.016581138595938683, -0.02675417996942997, 0.03309144824743271], "r_norm": [4.897111415863037, 4.8262786865234375, 4.823426723480225], "ldr": {"dL_sdil": -0.0021238327026367188, "dL_bp": -0.21677780151367188, "ratio": 0.009797279462227462}}, {"step": 100, "epoch": 0, "train_loss": 0.354571670293808, "cos_r_negg": [0.40948987007141113, 0.4940800368785858, 0.7221688032150269], "cos_apical_negg": [0.1771671026945114, 0.2853664755821228, 0.6299550533294678], "cos_Ac_negg": [0.4092680811882019, 0.4940105378627777, 0.72206711769104], "r_norm": [1.181720495223999, 1.1689503192901611, 1.1525987386703491]}, {"step": 200, "epoch": 0, "train_loss": 0.4218946099281311, "cos_r_negg": [0.4224170446395874, 0.4905916750431061, 0.7656741738319397], "cos_apical_negg": [0.12543192505836487, 0.20883363485336304, 0.679609477519989], "cos_Ac_negg": [0.42178788781166077, 0.4902670979499817, 0.7654527425765991], "r_norm": [0.9258381128311157, 0.9121930599212646, 0.9058879017829895]}, {"step": 300, "epoch": 0, "train_loss": 0.41522276401519775, "cos_r_negg": [0.4006715416908264, 0.46267831325531006, 0.8000354766845703], "cos_apical_negg": [0.12734946608543396, 0.25871264934539795, 0.6939327716827393], "cos_Ac_negg": [0.4005220830440521, 0.4625197649002075, 0.7999415993690491], "r_norm": [0.7790969610214233, 0.7675883769989014, 0.7631418704986572]}, {"step": 400, "epoch": 0, "train_loss": 0.22118234634399414, "cos_r_negg": [0.41696903109550476, 0.4606304168701172, 0.6738429069519043], "cos_apical_negg": [0.11484207212924957, 0.1932595819234848, 0.602165162563324], "cos_Ac_negg": [0.4168544411659241, 0.4607000946998596, 0.6736845970153809], "r_norm": [0.6956605911254883, 0.6850021481513977, 0.6820066571235657]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9206, "test_loss": 0.311843473815918}, {"step": 500, "epoch": 1, "train_loss": 0.3052837550640106, "cos_r_negg": [0.3626522123813629, 0.4173111617565155, 0.5963093042373657], "cos_apical_negg": [0.07664995640516281, 0.18701238930225372, 0.6576422452926636], "cos_Ac_negg": [0.36269712448120117, 0.41732293367385864, 0.5960298776626587], "r_norm": [0.6405333280563354, 0.6288838386535645, 0.6260596513748169], "ldr": {"dL_sdil": -0.014568626880645752, "dL_bp": -0.014336705207824707, "ratio": 1.0161767762857024}}, {"step": 600, "epoch": 1, "train_loss": 0.16827498376369476, "cos_r_negg": [0.37631213665008545, 0.44294190406799316, 0.5625686645507812], "cos_apical_negg": [0.10781636089086533, 0.1685357689857483, 0.5388386845588684], "cos_Ac_negg": [0.3762630224227905, 0.442885160446167, 0.5623227953910828], "r_norm": [0.6341857314109802, 0.6270827651023865, 0.624110758304596]}, {"step": 700, "epoch": 1, "train_loss": 0.21706843376159668, "cos_r_negg": [0.38124769926071167, 0.43224507570266724, 0.6496527194976807], "cos_apical_negg": [0.1277361810207367, 0.16795147955417633, 0.5623214244842529], "cos_Ac_negg": [0.3811199963092804, 0.4322311282157898, 0.6495639085769653], "r_norm": [0.6162890195846558, 0.6032637357711792, 0.6019018888473511]}, {"step": 800, "epoch": 1, "train_loss": 0.229483500123024, "cos_r_negg": [0.3993667960166931, 0.4717196226119995, 0.6010404825210571], "cos_apical_negg": [0.10239509493112564, 0.16297876834869385, 0.546845555305481], "cos_Ac_negg": [0.3992767930030823, 0.4717075824737549, 0.6010348796844482], "r_norm": [0.5413546562194824, 0.5288864970207214, 0.5284166932106018]}, {"step": 900, "epoch": 1, "train_loss": 0.32718586921691895, "cos_r_negg": [0.39117228984832764, 0.4628865420818329, 0.6911308765411377], "cos_apical_negg": [0.111225426197052, 0.17997516691684723, 0.6296118497848511], "cos_Ac_negg": [0.39112961292266846, 0.4628659784793854, 0.6910473108291626], "r_norm": [0.5905939936637878, 0.579919695854187, 0.5788048505783081]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9394, "test_loss": 0.23626662445068358}, {"step": 1000, "epoch": 2, "train_loss": 0.07668660581111908, "cos_r_negg": [0.3927341103553772, 0.5061125755310059, 0.5423319339752197], "cos_apical_negg": [0.09865610301494598, 0.18970930576324463, 0.4494820237159729], "cos_Ac_negg": [0.3927924335002899, 0.5060699582099915, 0.5423330664634705], "r_norm": [0.5557097792625427, 0.5447742938995361, 0.5418888330459595], "ldr": {"dL_sdil": -0.013394027948379517, "dL_bp": -0.013386756181716919, "ratio": 1.0005432060286965}}, {"step": 1100, "epoch": 2, "train_loss": 0.3958793878555298, "cos_r_negg": [0.3528103530406952, 0.4424353539943695, 0.5622879862785339], "cos_apical_negg": [0.09641429036855698, 0.1834137737751007, 0.5174195170402527], "cos_Ac_negg": [0.35279470682144165, 0.44244295358657837, 0.5621176958084106], "r_norm": [0.5224853754043579, 0.5165130496025085, 0.5115327835083008]}, {"step": 1200, "epoch": 2, "train_loss": 0.28291386365890503, "cos_r_negg": [0.36777395009994507, 0.45954036712646484, 0.629369854927063], "cos_apical_negg": [0.08748626708984375, 0.20882287621498108, 0.5821536779403687], "cos_Ac_negg": [0.36777278780937195, 0.45949238538742065, 0.6293504238128662], "r_norm": [0.48233214020729065, 0.4738706350326538, 0.4739091396331787]}, {"step": 1300, "epoch": 2, "train_loss": 0.23181961476802826, "cos_r_negg": [0.3398638963699341, 0.4273473620414734, 0.49137306213378906], "cos_apical_negg": [0.07635918259620667, 0.1942366063594818, 0.5700696706771851], "cos_Ac_negg": [0.3398456275463104, 0.4272955060005188, 0.4912376403808594], "r_norm": [0.5011234283447266, 0.4903164505958557, 0.49216747283935547]}, {"step": 1400, "epoch": 2, "train_loss": 0.24559831619262695, "cos_r_negg": [0.3553113639354706, 0.4113725423812866, 0.48809611797332764], "cos_apical_negg": [0.08183115720748901, 0.16185510158538818, 0.5142189860343933], "cos_Ac_negg": [0.3552836775779724, 0.41137421131134033, 0.4880104064941406], "r_norm": [0.4710192084312439, 0.46069324016571045, 0.46100401878356934]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9468, "test_loss": 0.21931175231933595}, {"step": 1500, "epoch": 3, "train_loss": 0.15713244676589966, "cos_r_negg": [0.3175938129425049, 0.39262649416923523, 0.4930814504623413], "cos_apical_negg": [0.09748604893684387, 0.15574148297309875, 0.5159343481063843], "cos_Ac_negg": [0.317560076713562, 0.3925871253013611, 0.49292829632759094], "r_norm": [0.4714347720146179, 0.46116870641708374, 0.4615328311920166], "ldr": {"dL_sdil": -0.010026350617408752, "dL_bp": -0.010028526186943054, "ratio": 0.9997830618882828}}, {"step": 1600, "epoch": 3, "train_loss": 0.14471080899238586, "cos_r_negg": [0.3618842661380768, 0.43481770157814026, 0.5564713478088379], "cos_apical_negg": [0.053075097501277924, 0.146455779671669, 0.5378377437591553], "cos_Ac_negg": [0.36187559366226196, 0.43482112884521484, 0.5563569068908691], "r_norm": [0.45724189281463623, 0.4426780045032501, 0.44519713521003723]}, {"step": 1700, "epoch": 3, "train_loss": 0.1199292540550232, "cos_r_negg": [0.3346858024597168, 0.41836243867874146, 0.5202815532684326], "cos_apical_negg": [0.086707204580307, 0.21298858523368835, 0.5306088924407959], "cos_Ac_negg": [0.3346349596977234, 0.4182862341403961, 0.5201213359832764], "r_norm": [0.4878188669681549, 0.4770936965942383, 0.4752395451068878]}, {"step": 1800, "epoch": 3, "train_loss": 0.06115349382162094, "cos_r_negg": [0.3116943836212158, 0.37237071990966797, 0.4174896478652954], "cos_apical_negg": [0.09862097352743149, 0.17975583672523499, 0.4274527430534363], "cos_Ac_negg": [0.3116356432437897, 0.37228354811668396, 0.41747552156448364], "r_norm": [0.5187429189682007, 0.5102837681770325, 0.5100627541542053]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9507, "test_loss": 0.19927916564941406}, {"step": 1900, "epoch": 4, "train_loss": 0.08558028936386108, "cos_r_negg": [0.3055054545402527, 0.3636631369590759, 0.5528383255004883], "cos_apical_negg": [0.06964286416769028, 0.16293182969093323, 0.4555685818195343], "cos_Ac_negg": [0.3055107593536377, 0.3636547923088074, 0.552701473236084], "r_norm": [0.39640918374061584, 0.38864558935165405, 0.3887024521827698]}, {"step": 2000, "epoch": 4, "train_loss": 0.20863178372383118, "cos_r_negg": [0.2909287214279175, 0.36311227083206177, 0.4310835003852844], "cos_apical_negg": [0.08341501653194427, 0.1085396558046341, 0.4226093590259552], "cos_Ac_negg": [0.2908599376678467, 0.36305713653564453, 0.43094033002853394], "r_norm": [0.4183720052242279, 0.40805768966674805, 0.40764540433883667], "ldr": {"dL_sdil": -0.008532971143722534, "dL_bp": -0.010096043348312378, "ratio": 0.8451797252979186}}, {"step": 2100, "epoch": 4, "train_loss": 0.11507221311330795, "cos_r_negg": [0.3011534810066223, 0.3769491910934448, 0.45562195777893066], "cos_apical_negg": [0.07453006505966187, 0.11422134935855865, 0.4654961824417114], "cos_Ac_negg": [0.3011004626750946, 0.3769018054008484, 0.4554436206817627], "r_norm": [0.4303979277610779, 0.4184738099575043, 0.42080533504486084]}, {"step": 2200, "epoch": 4, "train_loss": 0.07536563277244568, "cos_r_negg": [0.2964628040790558, 0.3720691502094269, 0.4125217795372009], "cos_apical_negg": [0.054394520819187164, 0.12011854350566864, 0.4078434705734253], "cos_Ac_negg": [0.2964456379413605, 0.3720472455024719, 0.4124046564102173], "r_norm": [0.39150893688201904, 0.3806436061859131, 0.3817109763622284]}, {"step": 2300, "epoch": 4, "train_loss": 0.3441089987754822, "cos_r_negg": [0.314182311296463, 0.395413339138031, 0.48691341280937195], "cos_apical_negg": [0.0562342144548893, 0.15732164680957794, 0.5054457187652588], "cos_Ac_negg": [0.31417709589004517, 0.39541733264923096, 0.4867471158504486], "r_norm": [0.37638723850250244, 0.3671910762786865, 0.36852917075157166]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.949, "test_loss": 0.2120886833190918}, {"step": 2400, "epoch": 5, "train_loss": 0.24819013476371765, "cos_r_negg": [0.29006025195121765, 0.38233426213264465, 0.4226503372192383], "cos_apical_negg": [0.05794958770275116, 0.16952604055404663, 0.43387433886528015], "cos_Ac_negg": [0.2900320291519165, 0.3822912573814392, 0.42244622111320496], "r_norm": [0.3746529519557953, 0.3633244037628174, 0.36511343717575073]}, {"step": 2500, "epoch": 5, "train_loss": 0.06795045733451843, "cos_r_negg": [0.28351694345474243, 0.36523333191871643, 0.4373854398727417], "cos_apical_negg": [0.06256628781557083, 0.16245101392269135, 0.44031962752342224], "cos_Ac_negg": [0.2835214138031006, 0.3652125597000122, 0.4372408092021942], "r_norm": [0.360385537147522, 0.3505025804042816, 0.3515985906124115], "ldr": {"dL_sdil": -0.007574334740638733, "dL_bp": -0.008766129612922668, "ratio": 0.8640454881562508}}, {"step": 2600, "epoch": 5, "train_loss": 0.1351993978023529, "cos_r_negg": [0.2961367964744568, 0.3669757843017578, 0.4170069098472595], "cos_apical_negg": [0.03270693123340607, 0.17077741026878357, 0.39031481742858887], "cos_Ac_negg": [0.29616183042526245, 0.36695927381515503, 0.4168417453765869], "r_norm": [0.3558770418167114, 0.34683164954185486, 0.3465604782104492]}, {"step": 2700, "epoch": 5, "train_loss": 0.1533799171447754, "cos_r_negg": [0.30687782168388367, 0.3699445128440857, 0.46407997608184814], "cos_apical_negg": [0.05205798149108887, 0.16318313777446747, 0.41084709763526917], "cos_Ac_negg": [0.3069007396697998, 0.36995577812194824, 0.46401360630989075], "r_norm": [0.3477948009967804, 0.3397001624107361, 0.33910202980041504]}, {"step": 2800, "epoch": 5, "train_loss": 0.23239493370056152, "cos_r_negg": [0.28342458605766296, 0.3175637125968933, 0.372060626745224], "cos_apical_negg": [0.05275917798280716, 0.09765138477087021, 0.31998056173324585], "cos_Ac_negg": [0.2833825945854187, 0.3175467848777771, 0.3720134496688843], "r_norm": [0.3499724268913269, 0.34066063165664673, 0.340814471244812]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9519, "test_loss": 0.20544991683959962}, {"step": 2900, "epoch": 6, "train_loss": 0.10849648714065552, "cos_r_negg": [0.2878199517726898, 0.3436349630355835, 0.47417017817497253], "cos_apical_negg": [0.05432596802711487, 0.11443020403385162, 0.3455365300178528], "cos_Ac_negg": [0.28781020641326904, 0.34365424513816833, 0.4741502106189728], "r_norm": [0.35208290815353394, 0.34019559621810913, 0.34312278032302856]}, {"step": 3000, "epoch": 6, "train_loss": 0.3311569392681122, "cos_r_negg": [0.28052449226379395, 0.35894688963890076, 0.40502479672431946], "cos_apical_negg": [0.035240545868873596, 0.14896342158317566, 0.42349112033843994], "cos_Ac_negg": [0.2805261015892029, 0.3588980436325073, 0.40476810932159424], "r_norm": [0.35302913188934326, 0.3448631763458252, 0.3449799418449402], "ldr": {"dL_sdil": -0.006956323981285095, "dL_bp": -0.00735849142074585, "ratio": 0.9453464825239966}}, {"step": 3100, "epoch": 6, "train_loss": 0.18019261956214905, "cos_r_negg": [0.2721954882144928, 0.33980417251586914, 0.4479500651359558], "cos_apical_negg": [0.02852373942732811, 0.17295899987220764, 0.35839468240737915], "cos_Ac_negg": [0.27219918370246887, 0.33978500962257385, 0.4477222263813019], "r_norm": [0.35866016149520874, 0.351005882024765, 0.34970420598983765]}, {"step": 3200, "epoch": 6, "train_loss": 0.199020117521286, "cos_r_negg": [0.25762081146240234, 0.3294498026371002, 0.42619872093200684], "cos_apical_negg": [0.04191029444336891, 0.1414119303226471, 0.40176644921302795], "cos_Ac_negg": [0.25764650106430054, 0.3294360637664795, 0.4260782301425934], "r_norm": [0.3657132685184479, 0.3539370894432068, 0.3553954064846039]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9549, "test_loss": 0.187177938079834}, {"step": 3300, "epoch": 7, "train_loss": 0.13309548795223236, "cos_r_negg": [0.27109068632125854, 0.3110350966453552, 0.3231617212295532], "cos_apical_negg": [0.023300375789403915, 0.11186415702104568, 0.3350473642349243], "cos_Ac_negg": [0.271141916513443, 0.3110065162181854, 0.3226667046546936], "r_norm": [0.3055375814437866, 0.29615113139152527, 0.2990162968635559]}, {"step": 3400, "epoch": 7, "train_loss": 0.25961431860923767, "cos_r_negg": [0.2954622507095337, 0.3442643880844116, 0.3928048610687256], "cos_apical_negg": [0.03685184568166733, 0.1230539083480835, 0.39098864793777466], "cos_Ac_negg": [0.29547008872032166, 0.34424111247062683, 0.3924611210823059], "r_norm": [0.31034815311431885, 0.30178070068359375, 0.3029753267765045]}, {"step": 3500, "epoch": 7, "train_loss": 0.11480915546417236, "cos_r_negg": [0.2152632176876068, 0.28108400106430054, 0.31355172395706177], "cos_apical_negg": [0.031719353049993515, 0.08576257526874542, 0.28526777029037476], "cos_Ac_negg": [0.21533070504665375, 0.28105199337005615, 0.3134344816207886], "r_norm": [0.3102531433105469, 0.302903413772583, 0.3034309148788452], "ldr": {"dL_sdil": -0.004531398415565491, "dL_bp": -0.006715118885040283, "ratio": 0.674805389620188}}, {"step": 3600, "epoch": 7, "train_loss": 0.0902668908238411, "cos_r_negg": [0.2389834225177765, 0.29262611269950867, 0.3521190881729126], "cos_apical_negg": [0.04988425225019455, 0.08719882369041443, 0.36276113986968994], "cos_Ac_negg": [0.2389843910932541, 0.29260218143463135, 0.3519917130470276], "r_norm": [0.3253469467163086, 0.313697874546051, 0.31717634201049805]}, {"step": 3700, "epoch": 7, "train_loss": 0.08515805006027222, "cos_r_negg": [0.2504104971885681, 0.32089632749557495, 0.3823220133781433], "cos_apical_negg": [0.024581369012594223, 0.08212266117334366, 0.3741797208786011], "cos_Ac_negg": [0.250422865152359, 0.32092148065567017, 0.3821236789226532], "r_norm": [0.33187663555145264, 0.3216373920440674, 0.3234902024269104]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9547, "test_loss": 0.19444280395507813}, {"step": 3800, "epoch": 8, "train_loss": 0.17694689333438873, "cos_r_negg": [0.25899192690849304, 0.31005844473838806, 0.3392488956451416], "cos_apical_negg": [0.03858417272567749, 0.0742853581905365, 0.3315388560295105], "cos_Ac_negg": [0.25906404852867126, 0.31005531549453735, 0.3390854001045227], "r_norm": [0.3168758153915405, 0.30747151374816895, 0.30865925550460815]}, {"step": 3900, "epoch": 8, "train_loss": 0.22353720664978027, "cos_r_negg": [0.22583681344985962, 0.2957781255245209, 0.31162646412849426], "cos_apical_negg": [0.029736997559666634, 0.1047627180814743, 0.21787340939044952], "cos_Ac_negg": [0.2258198857307434, 0.2957325577735901, 0.31153881549835205], "r_norm": [0.3253486156463623, 0.31517934799194336, 0.3172367215156555]}, {"step": 4000, "epoch": 8, "train_loss": 0.17737451195716858, "cos_r_negg": [0.2658914029598236, 0.31257694959640503, 0.3707655668258667], "cos_apical_negg": [0.02327217161655426, 0.09213034063577652, 0.3644936680793762], "cos_Ac_negg": [0.26588621735572815, 0.31257036328315735, 0.3706303536891937], "r_norm": [0.316942423582077, 0.30696213245391846, 0.3082296550273895], "ldr": {"dL_sdil": -0.005902528762817383, "dL_bp": -0.007494226098060608, "ratio": 0.7876101795680901}}, {"step": 4100, "epoch": 8, "train_loss": 0.10142381489276886, "cos_r_negg": [0.24553251266479492, 0.29517728090286255, 0.3253936171531677], "cos_apical_negg": [0.05160326510667801, 0.1083495244383812, 0.3625643253326416], "cos_Ac_negg": [0.24549433588981628, 0.2951638698577881, 0.3251679837703705], "r_norm": [0.312822550535202, 0.3023613691329956, 0.30564385652542114]}, {"step": 4200, "epoch": 8, "train_loss": 0.2199912816286087, "cos_r_negg": [0.21222490072250366, 0.2799052596092224, 0.3014346659183502], "cos_apical_negg": [0.028287524357438087, 0.08523555099964142, 0.2326328158378601], "cos_Ac_negg": [0.21220120787620544, 0.27988097071647644, 0.30123424530029297], "r_norm": [0.29305383563041687, 0.2860552668571472, 0.2874717116355896]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9564, "test_loss": 0.18425435104370116}, {"step": 4300, "epoch": 9, "train_loss": 0.06437113136053085, "cos_r_negg": [0.2358243763446808, 0.28641313314437866, 0.35599151253700256], "cos_apical_negg": [0.013880165293812752, 0.11917011439800262, 0.40688377618789673], "cos_Ac_negg": [0.23583494126796722, 0.28636664152145386, 0.3556973934173584], "r_norm": [0.31460991501808167, 0.3027815818786621, 0.3047682046890259]}, {"step": 4400, "epoch": 9, "train_loss": 0.06619978696107864, "cos_r_negg": [0.22984090447425842, 0.28721553087234497, 0.3278498947620392], "cos_apical_negg": [-0.011740047484636307, 0.06577267497777939, 0.32726940512657166], "cos_Ac_negg": [0.22985759377479553, 0.2872340679168701, 0.32772067189216614], "r_norm": [0.27921512722969055, 0.27059951424598694, 0.2719123363494873]}, {"step": 4500, "epoch": 9, "train_loss": 0.21131262183189392, "cos_r_negg": [0.2579497694969177, 0.3100973963737488, 0.30231156945228577], "cos_apical_negg": [0.04233098775148392, 0.08674807846546173, 0.28760814666748047], "cos_Ac_negg": [0.25795358419418335, 0.310102254152298, 0.30227792263031006], "r_norm": [0.3042036294937134, 0.2929670512676239, 0.29672420024871826], "ldr": {"dL_sdil": -0.004691421985626221, "dL_bp": -0.006736353039741516, "ratio": 0.6964335090439734}}, {"step": 4600, "epoch": 9, "train_loss": 0.05833065137267113, "cos_r_negg": [0.23096181452274323, 0.28671765327453613, 0.3423420190811157], "cos_apical_negg": [0.03894319757819176, 0.08784064650535583, 0.22024700045585632], "cos_Ac_negg": [0.2309252768754959, 0.28670844435691833, 0.34227150678634644], "r_norm": [0.2815912961959839, 0.2706446051597595, 0.2737721800804138]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9552, "test_loss": 0.19371358871459962}, {"step": 4700, "epoch": 10, "train_loss": 0.15056955814361572, "cos_r_negg": [0.23341451585292816, 0.275524377822876, 0.30159619450569153], "cos_apical_negg": [0.045774202793836594, 0.07596422731876373, 0.25025731325149536], "cos_Ac_negg": [0.23339876532554626, 0.27553778886795044, 0.3014761209487915], "r_norm": [0.29377245903015137, 0.28158220648765564, 0.28575533628463745]}, {"step": 4800, "epoch": 10, "train_loss": 0.09141849726438522, "cos_r_negg": [0.22750751674175262, 0.2827100157737732, 0.3121715784072876], "cos_apical_negg": [0.040812667459249496, 0.06755371391773224, 0.2866986393928528], "cos_Ac_negg": [0.22749042510986328, 0.2826702296733856, 0.3121235966682434], "r_norm": [0.2803487777709961, 0.2702489197254181, 0.274594247341156]}, {"step": 4900, "epoch": 10, "train_loss": 0.0935787707567215, "cos_r_negg": [0.23496471345424652, 0.27388620376586914, 0.3047889769077301], "cos_apical_negg": [0.03285589814186096, 0.08217950165271759, 0.22296622395515442], "cos_Ac_negg": [0.2349862903356552, 0.2738833725452423, 0.30466389656066895], "r_norm": [0.27926450967788696, 0.26818063855171204, 0.2710397243499756]}, {"step": 5000, "epoch": 10, "train_loss": 0.038625895977020264, "cos_r_negg": [0.23448936641216278, 0.29541683197021484, 0.3449649512767792], "cos_apical_negg": [0.026775607839226723, 0.07217676937580109, 0.31250983476638794], "cos_Ac_negg": [0.23451963067054749, 0.295479953289032, 0.3446601927280426], "r_norm": [0.28283730149269104, 0.2720959186553955, 0.2745661735534668], "ldr": {"dL_sdil": -0.005496546626091003, "dL_bp": -0.009115532040596008, "ratio": 0.6029869240338513}}, {"step": 5100, "epoch": 10, "train_loss": 0.21542933583259583, "cos_r_negg": [0.22495007514953613, 0.27443286776542664, 0.28965920209884644], "cos_apical_negg": [0.0038940533995628357, 0.06687778979539871, 0.28868257999420166], "cos_Ac_negg": [0.22494767606258392, 0.2744307219982147, 0.28959786891937256], "r_norm": [0.2665850520133972, 0.2567015588283539, 0.2595881223678589]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9589, "test_loss": 0.18391245193481445}, {"step": 5200, "epoch": 11, "train_loss": 0.03786221891641617, "cos_r_negg": [0.22902637720108032, 0.27354294061660767, 0.3025188446044922], "cos_apical_negg": [0.051290445029735565, 0.06759743392467499, 0.20606330037117004], "cos_Ac_negg": [0.22904717922210693, 0.2735593914985657, 0.3024744987487793], "r_norm": [0.26560336351394653, 0.25717055797576904, 0.2590850591659546]}, {"step": 5300, "epoch": 11, "train_loss": 0.10808123648166656, "cos_r_negg": [0.22896569967269897, 0.2585393786430359, 0.2974613308906555], "cos_apical_negg": [0.0072798095643520355, 0.06636877357959747, 0.2188795506954193], "cos_Ac_negg": [0.22902745008468628, 0.2586253881454468, 0.2974051833152771], "r_norm": [0.2733529806137085, 0.26100292801856995, 0.26502537727355957]}, {"step": 5400, "epoch": 11, "train_loss": 0.199072927236557, "cos_r_negg": [0.2233084738254547, 0.26314419507980347, 0.2899826467037201], "cos_apical_negg": [0.00771000050008297, 0.07246391475200653, 0.22055357694625854], "cos_Ac_negg": [0.22329072654247284, 0.2631770372390747, 0.2899090647697449], "r_norm": [0.30120620131492615, 0.28904008865356445, 0.2931087613105774]}, {"step": 5500, "epoch": 11, "train_loss": 0.20055337250232697, "cos_r_negg": [0.21133768558502197, 0.23950855433940887, 0.2701360583305359], "cos_apical_negg": [0.04376184940338135, 0.03876647353172302, 0.18125934898853302], "cos_Ac_negg": [0.21131333708763123, 0.2395479828119278, 0.27007484436035156], "r_norm": [0.29053816199302673, 0.2780545949935913, 0.2816965579986572], "ldr": {"dL_sdil": -0.004450201988220215, "dL_bp": -0.00824826955795288, "ratio": 0.5395315898629167}}, {"step": 5600, "epoch": 11, "train_loss": 0.126506969332695, "cos_r_negg": [0.23953893780708313, 0.2824479639530182, 0.30662205815315247], "cos_apical_negg": [0.03706660866737366, 0.10186675935983658, 0.2106373906135559], "cos_Ac_negg": [0.23959125578403473, 0.28245654702186584, 0.30653202533721924], "r_norm": [0.30468881130218506, 0.2926936447620392, 0.29458853602409363]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9579, "test_loss": 0.19195390396118164}, {"step": 5700, "epoch": 12, "train_loss": 0.08967877179384232, "cos_r_negg": [0.21077533066272736, 0.23948656022548676, 0.2750556766986847], "cos_apical_negg": [0.02860533446073532, 0.0595209114253521, 0.21731853485107422], "cos_Ac_negg": [0.21073806285858154, 0.23946264386177063, 0.2751821279525757], "r_norm": [0.28939589858055115, 0.27771109342575073, 0.28099897503852844]}, {"step": 5800, "epoch": 12, "train_loss": 0.05890396609902382, "cos_r_negg": [0.21447233855724335, 0.24529846012592316, 0.30949723720550537], "cos_apical_negg": [0.0038862908259034157, 0.04426558315753937, 0.28895819187164307], "cos_Ac_negg": [0.21441778540611267, 0.24530361592769623, 0.30945920944213867], "r_norm": [0.25535374879837036, 0.2450881004333496, 0.24786671996116638]}, {"step": 5900, "epoch": 12, "train_loss": 0.05154634267091751, "cos_r_negg": [0.19446277618408203, 0.22660233080387115, 0.2689710259437561], "cos_apical_negg": [0.010688330978155136, 0.042043544352054596, 0.2132973074913025], "cos_Ac_negg": [0.1944444626569748, 0.22659774124622345, 0.2688741683959961], "r_norm": [0.24351239204406738, 0.23183901607990265, 0.23559917509555817]}, {"step": 6000, "epoch": 12, "train_loss": 0.026518069207668304, "cos_r_negg": [0.22935675084590912, 0.2517244517803192, 0.26853829622268677], "cos_apical_negg": [0.012940449640154839, 0.04862320423126221, 0.22226621210575104], "cos_Ac_negg": [0.2293858379125595, 0.25175678730010986, 0.2686704397201538], "r_norm": [0.264340341091156, 0.2518172264099121, 0.25426769256591797], "ldr": {"dL_sdil": -0.006428390741348267, "dL_bp": -0.010369852185249329, "ratio": 0.619911511418878}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9584, "test_loss": 0.1835839588165283}, {"step": 6100, "epoch": 13, "train_loss": 0.06954273581504822, "cos_r_negg": [0.194825679063797, 0.24101389944553375, 0.30091187357902527], "cos_apical_negg": [0.00934390164911747, 0.07383058220148087, 0.25413602590560913], "cos_Ac_negg": [0.19485333561897278, 0.241000697016716, 0.3010340631008148], "r_norm": [0.2561012804508209, 0.2447504848241806, 0.24863633513450623]}, {"step": 6200, "epoch": 13, "train_loss": 0.04477343335747719, "cos_r_negg": [0.1878224015235901, 0.2199847251176834, 0.24407532811164856], "cos_apical_negg": [0.02219512313604355, 0.01871546171605587, 0.2027413696050644], "cos_Ac_negg": [0.1878327578306198, 0.21998947858810425, 0.2440405786037445], "r_norm": [0.25637736916542053, 0.2448316216468811, 0.2472589612007141]}, {"step": 6300, "epoch": 13, "train_loss": 0.06889718770980835, "cos_r_negg": [0.200616717338562, 0.24257710576057434, 0.28241243958473206], "cos_apical_negg": [0.06119794026017189, 0.045447248965501785, 0.23216712474822998], "cos_Ac_negg": [0.20062166452407837, 0.24265258014202118, 0.2824299931526184], "r_norm": [0.26680195331573486, 0.2550649046897888, 0.25872063636779785]}, {"step": 6400, "epoch": 13, "train_loss": 0.05969580262899399, "cos_r_negg": [0.210836261510849, 0.23148703575134277, 0.2838355004787445], "cos_apical_negg": [0.018764115869998932, 0.05549445375800133, 0.20234736800193787], "cos_Ac_negg": [0.21076911687850952, 0.2315089851617813, 0.28407302498817444], "r_norm": [0.2894783020019531, 0.27576225996017456, 0.27874502539634705]}, {"step": 6500, "epoch": 13, "train_loss": 0.042694855481386185, "cos_r_negg": [0.22249245643615723, 0.2611536681652069, 0.29595276713371277], "cos_apical_negg": [0.011676089838147163, 0.04506925493478775, 0.2274012267589569], "cos_Ac_negg": [0.22252732515335083, 0.26120078563690186, 0.2958718538284302], "r_norm": [0.2742013931274414, 0.26220232248306274, 0.26537418365478516], "ldr": {"dL_sdil": -0.005292832851409912, "dL_bp": -0.008661419153213501, "ratio": 0.6110814818892816}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9584, "test_loss": 0.194653182220459}, {"step": 6600, "epoch": 14, "train_loss": 0.04261435940861702, "cos_r_negg": [0.1855010986328125, 0.21898628771305084, 0.287163108587265], "cos_apical_negg": [0.00647738017141819, 0.046435341238975525, 0.21486365795135498], "cos_Ac_negg": [0.18545223772525787, 0.21898266673088074, 0.2869402766227722], "r_norm": [0.2575647234916687, 0.24623623490333557, 0.24928104877471924]}, {"step": 6700, "epoch": 14, "train_loss": 0.07799383997917175, "cos_r_negg": [0.1735231578350067, 0.18533873558044434, 0.30120450258255005], "cos_apical_negg": [0.04645295441150665, 0.053029824048280716, 0.24667207896709442], "cos_Ac_negg": [0.17348648607730865, 0.1852864772081375, 0.3009122610092163], "r_norm": [0.2859070599079132, 0.2738664150238037, 0.2766638994216919]}, {"step": 6800, "epoch": 14, "train_loss": 0.192641943693161, "cos_r_negg": [0.1949756145477295, 0.24111878871917725, 0.2974018454551697], "cos_apical_negg": [0.0397103875875473, 0.0335024930536747, 0.16501350700855255], "cos_Ac_negg": [0.19497132301330566, 0.24115285277366638, 0.2973495125770569], "r_norm": [0.27731722593307495, 0.26534324884414673, 0.26979339122772217]}, {"step": 6900, "epoch": 14, "train_loss": 0.05707872658967972, "cos_r_negg": [0.17775830626487732, 0.23034465312957764, 0.2533235549926758], "cos_apical_negg": [0.005555080249905586, 0.06054583191871643, 0.1894989162683487], "cos_Ac_negg": [0.1777692586183548, 0.23038877546787262, 0.25332021713256836], "r_norm": [0.2580033242702484, 0.24572646617889404, 0.24875102937221527]}, {"step": 7000, "epoch": 14, "train_loss": 0.13115863502025604, "cos_r_negg": [0.19121834635734558, 0.23183977603912354, 0.26908716559410095], "cos_apical_negg": [0.02594858966767788, 0.0576966255903244, 0.20463576912879944], "cos_Ac_negg": [0.1912306696176529, 0.23184677958488464, 0.269147664308548], "r_norm": [0.2929958403110504, 0.2797738015651703, 0.2834494113922119], "ldr": {"dL_sdil": -0.004326134920120239, "dL_bp": -0.009723544120788574, "ratio": 0.4449133840891537}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.956, "test_loss": 0.2118159149169922}], "final": {"test_acc": 0.956, "test_loss": 0.2118159149169922, "wall_s": 41.38318920135498, "cos_r_negg": [0.19462116062641144, 0.23295272886753082, 0.27229419350624084], "cos_apical_negg": [0.04181640222668648, 0.08553440868854523, 0.18258577585220337], "cos_Ac_negg": [0.19462744891643524, 0.232876718044281, 0.2721676826477051]}}
\ No newline at end of file diff --git a/results/diag_nuis_low_rho0p2_res1_s0.json b/results/diag_nuis_low_rho0p2_res1_s0.json new file mode 100644 index 0000000..7e1b4df --- /dev/null +++ b/results/diag_nuis_low_rho0p2_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.2, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_low_rho0p2_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.019516926258802414, -0.02552565187215805, 0.03376537561416626], "cos_apical_negg": [-0.01901330053806305, -0.025642871856689453, 0.030623063445091248], "cos_Ac_negg": [-0.01951673999428749, -0.0255256500095129, 0.03376534581184387], "r_norm": [4.898183345794678, 4.826805114746094, 4.823857307434082], "ldr": {"dL_sdil": -0.009201765060424805, "dL_bp": -0.24068951606750488, "ratio": 0.03823085114286422}}, {"step": 100, "epoch": 0, "train_loss": 0.20720425248146057, "cos_r_negg": [0.4560698866844177, 0.3799092173576355, 0.3596936762332916], "cos_apical_negg": [0.11800754070281982, 0.1147136241197586, 0.16613300144672394], "cos_Ac_negg": [0.4572519063949585, 0.381181538105011, 0.36056190729141235], "r_norm": [0.8240674734115601, 0.8074890971183777, 0.805279016494751]}, {"step": 200, "epoch": 0, "train_loss": 0.32886016368865967, "cos_r_negg": [0.4721740484237671, 0.40303605794906616, 0.38011831045150757], "cos_apical_negg": [0.10742950439453125, 0.11374245584011078, 0.15492889285087585], "cos_Ac_negg": [0.4744020104408264, 0.40578994154930115, 0.3833943009376526], "r_norm": [0.7475301623344421, 0.7304760217666626, 0.7292832136154175]}, {"step": 300, "epoch": 0, "train_loss": 0.1827433556318283, "cos_r_negg": [0.49058136343955994, 0.4557255506515503, 0.41817498207092285], "cos_apical_negg": [0.09387607872486115, 0.09999481588602066, 0.14982113242149353], "cos_Ac_negg": [0.4924466609954834, 0.4564589858055115, 0.4186569154262543], "r_norm": [0.5988276600837708, 0.5881868004798889, 0.5856267213821411]}, {"step": 400, "epoch": 0, "train_loss": 0.19849328696727753, "cos_r_negg": [0.5320627689361572, 0.4917117953300476, 0.45732900500297546], "cos_apical_negg": [0.09529949724674225, 0.10565420985221863, 0.14770174026489258], "cos_Ac_negg": [0.5334955453872681, 0.4921759366989136, 0.4570450782775879], "r_norm": [0.5587079524993896, 0.5461689829826355, 0.544512927532196]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9444, "test_loss": 0.18775836486816405}, {"step": 500, "epoch": 1, "train_loss": 0.22689279913902283, "cos_r_negg": [0.518352746963501, 0.4924679100513458, 0.4760664105415344], "cos_apical_negg": [0.08258219063282013, 0.09276214241981506, 0.14235663414001465], "cos_Ac_negg": [0.5208925008773804, 0.49402713775634766, 0.4768965542316437], "r_norm": [0.5135685205459595, 0.49644920229911804, 0.4960799813270569], "ldr": {"dL_sdil": -0.012973681092262268, "dL_bp": -0.011488854885101318, "ratio": 1.129240574627369}}, {"step": 600, "epoch": 1, "train_loss": 0.0776064544916153, "cos_r_negg": [0.5252289772033691, 0.502976655960083, 0.4807737469673157], "cos_apical_negg": [0.08637966960668564, 0.08884815871715546, 0.14246749877929688], "cos_Ac_negg": [0.5288116931915283, 0.5047304630279541, 0.4826890826225281], "r_norm": [0.47395291924476624, 0.46290624141693115, 0.4614240527153015]}, {"step": 700, "epoch": 1, "train_loss": 0.21897561848163605, "cos_r_negg": [0.5115370750427246, 0.5047791600227356, 0.4958755373954773], "cos_apical_negg": [0.06713128089904785, 0.08597706258296967, 0.14841443300247192], "cos_Ac_negg": [0.5136685371398926, 0.5062215328216553, 0.49651795625686646], "r_norm": [0.4447997808456421, 0.43120214343070984, 0.43148308992385864]}, {"step": 800, "epoch": 1, "train_loss": 0.11325228214263916, "cos_r_negg": [0.5302717089653015, 0.5298453569412231, 0.5081886649131775], "cos_apical_negg": [0.06364849209785461, 0.09421849250793457, 0.1592203676700592], "cos_Ac_negg": [0.5326454043388367, 0.5304535031318665, 0.5076013803482056], "r_norm": [0.39312827587127686, 0.3832486867904663, 0.38258886337280273]}, {"step": 900, "epoch": 1, "train_loss": 0.18200311064720154, "cos_r_negg": [0.5702155828475952, 0.5531561374664307, 0.5118391513824463], "cos_apical_negg": [0.07652077078819275, 0.10256987810134888, 0.15148979425430298], "cos_Ac_negg": [0.5709106922149658, 0.5534185767173767, 0.5113274455070496], "r_norm": [0.4159635305404663, 0.4055224061012268, 0.4036968946456909]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9543, "test_loss": 0.1573215847015381}, {"step": 1000, "epoch": 2, "train_loss": 0.04643802344799042, "cos_r_negg": [0.5445017218589783, 0.5341522693634033, 0.5277763605117798], "cos_apical_negg": [0.051489848643541336, 0.07223533093929291, 0.14561563730239868], "cos_Ac_negg": [0.5462212562561035, 0.5358578562736511, 0.5267815589904785], "r_norm": [0.355496346950531, 0.34455305337905884, 0.34347406029701233], "ldr": {"dL_sdil": -0.009742259979248047, "dL_bp": -0.009028464555740356, "ratio": 1.0790605555463864}}, {"step": 1100, "epoch": 2, "train_loss": 0.10056138038635254, "cos_r_negg": [0.5322949290275574, 0.547164797782898, 0.5300918817520142], "cos_apical_negg": [0.057358257472515106, 0.08037659525871277, 0.14548909664154053], "cos_Ac_negg": [0.5341708660125732, 0.5480469465255737, 0.5287167429924011], "r_norm": [0.3589257597923279, 0.3457219898700714, 0.3480779528617859]}, {"step": 1200, "epoch": 2, "train_loss": 0.11795825511217117, "cos_r_negg": [0.5263893604278564, 0.5441412925720215, 0.5197768211364746], "cos_apical_negg": [0.05794861912727356, 0.07501484453678131, 0.1442803591489792], "cos_Ac_negg": [0.5268850922584534, 0.5448527336120605, 0.5185595750808716], "r_norm": [0.34627944231033325, 0.33819010853767395, 0.3388124108314514]}, {"step": 1300, "epoch": 2, "train_loss": 0.10605527460575104, "cos_r_negg": [0.5358836650848389, 0.5459455251693726, 0.51361083984375], "cos_apical_negg": [0.04015921801328659, 0.07396218180656433, 0.13252627849578857], "cos_Ac_negg": [0.5371134281158447, 0.5457530617713928, 0.5116366744041443], "r_norm": [0.3273528516292572, 0.3196491003036499, 0.31749486923217773]}, {"step": 1400, "epoch": 2, "train_loss": 0.15307524800300598, "cos_r_negg": [0.5064762830734253, 0.5161098837852478, 0.4902312159538269], "cos_apical_negg": [0.03497637063264847, 0.07718580961227417, 0.1275871992111206], "cos_Ac_negg": [0.5067011713981628, 0.5157096982002258, 0.48784637451171875], "r_norm": [0.29957571625709534, 0.29161521792411804, 0.2900125980377197]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9661, "test_loss": 0.1204852825164795}, {"step": 1500, "epoch": 3, "train_loss": 0.0698956698179245, "cos_r_negg": [0.4961811900138855, 0.5310268998146057, 0.4978474974632263], "cos_apical_negg": [0.030895758420228958, 0.073004350066185, 0.12490352243185043], "cos_Ac_negg": [0.4969940185546875, 0.5304210782051086, 0.4955146312713623], "r_norm": [0.2544264495372772, 0.24644877016544342, 0.24746359884738922], "ldr": {"dL_sdil": -0.005140930414199829, "dL_bp": -0.00506901741027832, "ratio": 1.0141867739052726}}, {"step": 1600, "epoch": 3, "train_loss": 0.0789014995098114, "cos_r_negg": [0.4994402825832367, 0.5197998285293579, 0.5130748748779297], "cos_apical_negg": [0.03602364659309387, 0.05989626422524452, 0.13508304953575134], "cos_Ac_negg": [0.5005179643630981, 0.5194783806800842, 0.5095216035842896], "r_norm": [0.25973770022392273, 0.251307874917984, 0.25205036997795105]}, {"step": 1700, "epoch": 3, "train_loss": 0.06682585179805756, "cos_r_negg": [0.4757058024406433, 0.5004146099090576, 0.47526443004608154], "cos_apical_negg": [0.04281626641750336, 0.07103821635246277, 0.13133926689624786], "cos_Ac_negg": [0.47632718086242676, 0.4999530613422394, 0.4728115499019623], "r_norm": [0.30378809571266174, 0.29386523365974426, 0.2947423458099365]}, {"step": 1800, "epoch": 3, "train_loss": 0.03949519991874695, "cos_r_negg": [0.4988632798194885, 0.5236339569091797, 0.49526819586753845], "cos_apical_negg": [0.04092167690396309, 0.06489545106887817, 0.13144882023334503], "cos_Ac_negg": [0.49944034218788147, 0.5231953859329224, 0.49136853218078613], "r_norm": [0.2662772536277771, 0.25937896966934204, 0.25859761238098145]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.962, "test_loss": 0.1305105968475342}, {"step": 1900, "epoch": 4, "train_loss": 0.04540172964334488, "cos_r_negg": [0.4961163401603699, 0.532311201095581, 0.5135544538497925], "cos_apical_negg": [0.028665177524089813, 0.06601957976818085, 0.1310112178325653], "cos_Ac_negg": [0.4973849654197693, 0.5321346521377563, 0.5103310346603394], "r_norm": [0.2822105884552002, 0.27646303176879883, 0.2750280797481537]}, {"step": 2000, "epoch": 4, "train_loss": 0.11028040945529938, "cos_r_negg": [0.43829062581062317, 0.4881932735443115, 0.4720665216445923], "cos_apical_negg": [0.03399401158094406, 0.058208029717206955, 0.12531375885009766], "cos_Ac_negg": [0.43930310010910034, 0.4880529046058655, 0.4675171971321106], "r_norm": [0.2239913046360016, 0.21864327788352966, 0.21820829808712006], "ldr": {"dL_sdil": -0.005510784685611725, "dL_bp": -0.0061350017786026, "ratio": 0.8982531520743819}}, {"step": 2100, "epoch": 4, "train_loss": 0.027807198464870453, "cos_r_negg": [0.455223947763443, 0.5091505646705627, 0.4864268898963928], "cos_apical_negg": [0.04054104909300804, 0.06618750840425491, 0.13444280624389648], "cos_Ac_negg": [0.45667532086372375, 0.5087003707885742, 0.4815230071544647], "r_norm": [0.24564141035079956, 0.2398286759853363, 0.23780688643455505]}, {"step": 2200, "epoch": 4, "train_loss": 0.013736401684582233, "cos_r_negg": [0.4327235817909241, 0.4789857268333435, 0.4565269947052002], "cos_apical_negg": [0.020444907248020172, 0.06148357316851616, 0.12413620948791504], "cos_Ac_negg": [0.4342272877693176, 0.47791120409965515, 0.4516263008117676], "r_norm": [0.22277282178401947, 0.2159545123577118, 0.21577337384223938]}, {"step": 2300, "epoch": 4, "train_loss": 0.1452348232269287, "cos_r_negg": [0.4345741868019104, 0.4666666090488434, 0.4470028877258301], "cos_apical_negg": [0.03594515472650528, 0.05967184528708458, 0.12620416283607483], "cos_Ac_negg": [0.4339560866355896, 0.46607810258865356, 0.44227349758148193], "r_norm": [0.23884662985801697, 0.23301084339618683, 0.2317212074995041]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9677, "test_loss": 0.1195300910949707}, {"step": 2400, "epoch": 5, "train_loss": 0.07576563209295273, "cos_r_negg": [0.4333113431930542, 0.4679281711578369, 0.44398391246795654], "cos_apical_negg": [0.017782431095838547, 0.05849715694785118, 0.12608438730239868], "cos_Ac_negg": [0.43362292647361755, 0.46755844354629517, 0.438922256231308], "r_norm": [0.21344420313835144, 0.20824608206748962, 0.20770011842250824]}, {"step": 2500, "epoch": 5, "train_loss": 0.03994230926036835, "cos_r_negg": [0.37777048349380493, 0.41896992921829224, 0.4019518792629242], "cos_apical_negg": [0.016743242740631104, 0.05408822000026703, 0.12522226572036743], "cos_Ac_negg": [0.3792899549007416, 0.41993248462677, 0.39697742462158203], "r_norm": [0.20185455679893494, 0.19803205132484436, 0.1976049691438675], "ldr": {"dL_sdil": -0.006245836615562439, "dL_bp": -0.007252521812915802, "ratio": 0.8611951506908139}}, {"step": 2600, "epoch": 5, "train_loss": 0.01891755685210228, "cos_r_negg": [0.3704303503036499, 0.40972834825515747, 0.3915012776851654], "cos_apical_negg": [0.018268905580043793, 0.0507926382124424, 0.12278301268815994], "cos_Ac_negg": [0.37078022956848145, 0.4095395803451538, 0.38652411103248596], "r_norm": [0.1959180384874344, 0.1910243034362793, 0.1898036003112793]}, {"step": 2700, "epoch": 5, "train_loss": 0.0255320742726326, "cos_r_negg": [0.3964291512966156, 0.4413040280342102, 0.42182010412216187], "cos_apical_negg": [0.019239600747823715, 0.049195513129234314, 0.1268845796585083], "cos_Ac_negg": [0.3959196209907532, 0.4408583641052246, 0.41549748182296753], "r_norm": [0.18055550754070282, 0.175320565700531, 0.17385819554328918]}, {"step": 2800, "epoch": 5, "train_loss": 0.04729419946670532, "cos_r_negg": [0.40382805466651917, 0.4451998770236969, 0.42320412397384644], "cos_apical_negg": [0.022398851811885834, 0.0480986014008522, 0.13043440878391266], "cos_Ac_negg": [0.40410560369491577, 0.4455016553401947, 0.41741615533828735], "r_norm": [0.2163838893175125, 0.209800124168396, 0.20773287117481232]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9694, "test_loss": 0.11945083370208741}, {"step": 2900, "epoch": 6, "train_loss": 0.006296555511653423, "cos_r_negg": [0.3533351421356201, 0.39574792981147766, 0.38194960355758667], "cos_apical_negg": [0.015517255291342735, 0.052250489592552185, 0.1331884264945984], "cos_Ac_negg": [0.355678915977478, 0.39656829833984375, 0.3765743672847748], "r_norm": [0.205510675907135, 0.19954802095890045, 0.1992124617099762]}, {"step": 3000, "epoch": 6, "train_loss": 0.03698936849832535, "cos_r_negg": [0.3363742530345917, 0.37725725769996643, 0.37123340368270874], "cos_apical_negg": [0.01019035093486309, 0.052982669323682785, 0.1354113519191742], "cos_Ac_negg": [0.33733493089675903, 0.37746450304985046, 0.3663625717163086], "r_norm": [0.1813351958990097, 0.17647740244865417, 0.17598849534988403], "ldr": {"dL_sdil": -0.0053081512451171875, "dL_bp": -0.006433658301830292, "ratio": 0.8250595533814856}}, {"step": 3100, "epoch": 6, "train_loss": 0.03997653350234032, "cos_r_negg": [0.32278597354888916, 0.3597638010978699, 0.3466194272041321], "cos_apical_negg": [0.006306661292910576, 0.05574684217572212, 0.13018211722373962], "cos_Ac_negg": [0.32433459162712097, 0.3602352738380432, 0.341050922870636], "r_norm": [0.2025657594203949, 0.1966540664434433, 0.19589850306510925]}, {"step": 3200, "epoch": 6, "train_loss": 0.015686307102441788, "cos_r_negg": [0.3026968836784363, 0.34910643100738525, 0.3440398573875427], "cos_apical_negg": [-0.00566835468634963, 0.04428114369511604, 0.12208204716444016], "cos_Ac_negg": [0.30505913496017456, 0.3494424819946289, 0.3391879200935364], "r_norm": [0.17451635003089905, 0.1704329550266266, 0.16918087005615234]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9675, "test_loss": 0.12933441181182861}, {"step": 3300, "epoch": 7, "train_loss": 0.016955068334937096, "cos_r_negg": [0.30280065536499023, 0.3464622497558594, 0.33933037519454956], "cos_apical_negg": [0.0028160554356873035, 0.05315080285072327, 0.12500843405723572], "cos_Ac_negg": [0.3054389953613281, 0.3472543954849243, 0.3350687026977539], "r_norm": [0.17989301681518555, 0.17568236589431763, 0.17468030750751495]}, {"step": 3400, "epoch": 7, "train_loss": 0.03091752529144287, "cos_r_negg": [0.2887415885925293, 0.32733845710754395, 0.3142198622226715], "cos_apical_negg": [0.012202640995383263, 0.05083329975605011, 0.12281153351068497], "cos_Ac_negg": [0.2909994125366211, 0.3274838626384735, 0.30981266498565674], "r_norm": [0.18245676159858704, 0.1776871383190155, 0.17711219191551208]}, {"step": 3500, "epoch": 7, "train_loss": 0.002526980359107256, "cos_r_negg": [0.2776307463645935, 0.31304797530174255, 0.30889493227005005], "cos_apical_negg": [0.010596306063234806, 0.04984480142593384, 0.12387808412313461], "cos_Ac_negg": [0.2805289924144745, 0.3140209913253784, 0.3047606348991394], "r_norm": [0.163944810628891, 0.15973219275474548, 0.15938334167003632], "ldr": {"dL_sdil": -0.005481712520122528, "dL_bp": -0.007461570203304291, "ratio": 0.734659377418308}}, {"step": 3600, "epoch": 7, "train_loss": 0.001926591619849205, "cos_r_negg": [0.2766410708427429, 0.3109179735183716, 0.3051733076572418], "cos_apical_negg": [0.01300154346972704, 0.041793517768383026, 0.12349347770214081], "cos_Ac_negg": [0.27783018350601196, 0.3109878897666931, 0.3008573651313782], "r_norm": [0.16562733054161072, 0.1622338891029358, 0.16067373752593994]}, {"step": 3700, "epoch": 7, "train_loss": 0.029019419103860855, "cos_r_negg": [0.27375733852386475, 0.3093430995941162, 0.30635905265808105], "cos_apical_negg": [-0.004757965449243784, 0.03974289074540138, 0.11225108802318573], "cos_Ac_negg": [0.27632540464401245, 0.30991142988204956, 0.3025987148284912], "r_norm": [0.133812814950943, 0.13092832267284393, 0.12926605343818665]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9726, "test_loss": 0.1197734540939331}, {"step": 3800, "epoch": 8, "train_loss": 0.020506305620074272, "cos_r_negg": [0.2609327435493469, 0.29622310400009155, 0.2937408983707428], "cos_apical_negg": [-0.0008132867515087128, 0.04493826627731323, 0.11635374277830124], "cos_Ac_negg": [0.2627362310886383, 0.297172486782074, 0.29046446084976196], "r_norm": [0.16088099777698517, 0.15714684128761292, 0.15592435002326965]}, {"step": 3900, "epoch": 8, "train_loss": 0.027562374249100685, "cos_r_negg": [0.24690283834934235, 0.287659615278244, 0.2854064702987671], "cos_apical_negg": [0.006630089599639177, 0.04292266070842743, 0.11482829600572586], "cos_Ac_negg": [0.24869242310523987, 0.2886280119419098, 0.281788170337677], "r_norm": [0.15215270221233368, 0.14906710386276245, 0.14724694192409515]}, {"step": 4000, "epoch": 8, "train_loss": 0.01519346795976162, "cos_r_negg": [0.21978797018527985, 0.251234233379364, 0.2517244815826416], "cos_apical_negg": [0.01181202381849289, 0.040936972945928574, 0.11274947226047516], "cos_Ac_negg": [0.22197610139846802, 0.2526814937591553, 0.24853599071502686], "r_norm": [0.1421610713005066, 0.1389172375202179, 0.13751095533370972], "ldr": {"dL_sdil": -0.00547616183757782, "dL_bp": -0.0079764723777771, "ratio": 0.6865393093862789}}, {"step": 4100, "epoch": 8, "train_loss": 0.0031640788074582815, "cos_r_negg": [0.23090822994709015, 0.2655733823776245, 0.2627096474170685], "cos_apical_negg": [0.007657879497855902, 0.03466641902923584, 0.10710987448692322], "cos_Ac_negg": [0.23260904848575592, 0.26691487431526184, 0.25958487391471863], "r_norm": [0.13521380722522736, 0.13214032351970673, 0.13036230206489563]}, {"step": 4200, "epoch": 8, "train_loss": 0.02157864160835743, "cos_r_negg": [0.22413602471351624, 0.2547035813331604, 0.25224021077156067], "cos_apical_negg": [0.004489130340516567, 0.03717212378978729, 0.10925235599279404], "cos_Ac_negg": [0.22527465224266052, 0.25566601753234863, 0.24910634756088257], "r_norm": [0.13585026562213898, 0.1329030990600586, 0.13123327493667603]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.972, "test_loss": 0.13100981311798096}, {"step": 4300, "epoch": 9, "train_loss": 0.00426859688013792, "cos_r_negg": [0.21622473001480103, 0.24608829617500305, 0.24620768427848816], "cos_apical_negg": [0.0030632754787802696, 0.03513254225254059, 0.10686928778886795], "cos_Ac_negg": [0.21805903315544128, 0.24759963154792786, 0.24335172772407532], "r_norm": [0.13569112122058868, 0.13284394145011902, 0.13094843924045563]}, {"step": 4400, "epoch": 9, "train_loss": 0.0023573762737214565, "cos_r_negg": [0.20723524689674377, 0.23606529831886292, 0.2372949719429016], "cos_apical_negg": [0.012994794175028801, 0.03374059498310089, 0.10754804313182831], "cos_Ac_negg": [0.20902681350708008, 0.23773640394210815, 0.23462677001953125], "r_norm": [0.1324518918991089, 0.12974298000335693, 0.12811225652694702]}, {"step": 4500, "epoch": 9, "train_loss": 0.024856390431523323, "cos_r_negg": [0.1990593671798706, 0.22667831182479858, 0.22731530666351318], "cos_apical_negg": [0.010719822719693184, 0.03881388157606125, 0.10649359226226807], "cos_Ac_negg": [0.20103803277015686, 0.2281455546617508, 0.22477087378501892], "r_norm": [0.1479368805885315, 0.14449582993984222, 0.14257672429084778], "ldr": {"dL_sdil": -0.0068184733390808105, "dL_bp": -0.011349000036716461, "ratio": 0.6007994816302388}}, {"step": 4600, "epoch": 9, "train_loss": 0.0006705457344651222, "cos_r_negg": [0.1997426152229309, 0.2290957272052765, 0.23006536066532135], "cos_apical_negg": [0.013574485667049885, 0.03767723962664604, 0.10938790440559387], "cos_Ac_negg": [0.2019762098789215, 0.23055006563663483, 0.22741809487342834], "r_norm": [0.15341885387897491, 0.15031732618808746, 0.14804190397262573]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9734, "test_loss": 0.13374080486297607}, {"step": 4700, "epoch": 10, "train_loss": 0.0015674801543354988, "cos_r_negg": [0.18646520376205444, 0.21135398745536804, 0.21139559149742126], "cos_apical_negg": [0.013829442672431469, 0.03718261048197746, 0.10361390560865402], "cos_Ac_negg": [0.18802279233932495, 0.21266856789588928, 0.20844444632530212], "r_norm": [0.1420573592185974, 0.13966426253318787, 0.13743090629577637]}, {"step": 4800, "epoch": 10, "train_loss": 0.0013977123890072107, "cos_r_negg": [0.18784549832344055, 0.2126714289188385, 0.21459941565990448], "cos_apical_negg": [0.012076596729457378, 0.03907722234725952, 0.09633399546146393], "cos_Ac_negg": [0.18977698683738708, 0.21397997438907623, 0.21251502633094788], "r_norm": [0.14872132241725922, 0.14574193954467773, 0.14366304874420166]}, {"step": 4900, "epoch": 10, "train_loss": 0.002537726191803813, "cos_r_negg": [0.18453532457351685, 0.20816902816295624, 0.2083142250776291], "cos_apical_negg": [0.013001528568565845, 0.03525445610284805, 0.09514307975769043], "cos_Ac_negg": [0.18653619289398193, 0.20991268754005432, 0.20608650147914886], "r_norm": [0.13913799822330475, 0.1360703855752945, 0.1341627538204193]}, {"step": 5000, "epoch": 10, "train_loss": 0.00029283654293976724, "cos_r_negg": [0.1790531873703003, 0.20236000418663025, 0.20367254316806793], "cos_apical_negg": [0.00915782991796732, 0.034670256078243256, 0.09333822876214981], "cos_Ac_negg": [0.180527925491333, 0.2040417641401291, 0.2014128565788269], "r_norm": [0.13948453962802887, 0.13676591217517853, 0.13499653339385986], "ldr": {"dL_sdil": -0.006759658455848694, "dL_bp": -0.011344537138938904, "ratio": 0.5958514105125446}}, {"step": 5100, "epoch": 10, "train_loss": 0.001166645553894341, "cos_r_negg": [0.1774941384792328, 0.19988617300987244, 0.20065250992774963], "cos_apical_negg": [0.010501018725335598, 0.03752711042761803, 0.09508220851421356], "cos_Ac_negg": [0.17864765226840973, 0.2008398175239563, 0.19785438477993011], "r_norm": [0.14630359411239624, 0.14384523034095764, 0.1423577517271042]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9727, "test_loss": 0.13959815769195558}, {"step": 5200, "epoch": 11, "train_loss": 0.0005057034431956708, "cos_r_negg": [0.17119407653808594, 0.1947094351053238, 0.1955377161502838], "cos_apical_negg": [0.01346607506275177, 0.036842238157987595, 0.09451436251401901], "cos_Ac_negg": [0.17239035665988922, 0.1957586407661438, 0.19309139251708984], "r_norm": [0.14516931772232056, 0.14239320158958435, 0.14052855968475342]}, {"step": 5300, "epoch": 11, "train_loss": 0.002276922110468149, "cos_r_negg": [0.16881020367145538, 0.19389797747135162, 0.1956392526626587], "cos_apical_negg": [0.008678173646330833, 0.034508123993873596, 0.08992566168308258], "cos_Ac_negg": [0.17071065306663513, 0.19545282423496246, 0.19359004497528076], "r_norm": [0.14157912135124207, 0.13896429538726807, 0.13739068806171417]}, {"step": 5400, "epoch": 11, "train_loss": 0.0024333116598427296, "cos_r_negg": [0.16428053379058838, 0.1873890906572342, 0.18808835744857788], "cos_apical_negg": [0.011569863185286522, 0.035289160907268524, 0.09052517265081406], "cos_Ac_negg": [0.16570982336997986, 0.18850667774677277, 0.18583758175373077], "r_norm": [0.14646819233894348, 0.14385995268821716, 0.14194801449775696]}, {"step": 5500, "epoch": 11, "train_loss": 0.0015745742712169886, "cos_r_negg": [0.16143254935741425, 0.18354007601737976, 0.18404385447502136], "cos_apical_negg": [0.015511251986026764, 0.0361345075070858, 0.09334743022918701], "cos_Ac_negg": [0.1628625988960266, 0.18454262614250183, 0.18176285922527313], "r_norm": [0.14970797300338745, 0.14724522829055786, 0.14508797228336334], "ldr": {"dL_sdil": -0.007584363222122192, "dL_bp": -0.012636572122573853, "ratio": 0.6001915035623908}}, {"step": 5600, "epoch": 11, "train_loss": 0.00026500236708670855, "cos_r_negg": [0.15747985243797302, 0.17864133417606354, 0.18018221855163574], "cos_apical_negg": [0.013418947346508503, 0.03440164774656296, 0.08656476438045502], "cos_Ac_negg": [0.15878228843212128, 0.17988315224647522, 0.1781887710094452], "r_norm": [0.13677451014518738, 0.13422688841819763, 0.1323801875114441]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9726, "test_loss": 0.1431348155975342}, {"step": 5700, "epoch": 12, "train_loss": 0.001291529624722898, "cos_r_negg": [0.1526879072189331, 0.17417368292808533, 0.17488881945610046], "cos_apical_negg": [0.014797704294323921, 0.0345303937792778, 0.08734732866287231], "cos_Ac_negg": [0.15434011816978455, 0.17518828809261322, 0.1727391481399536], "r_norm": [0.14300128817558289, 0.14052969217300415, 0.13872411847114563]}, {"step": 5800, "epoch": 12, "train_loss": 0.00039351428858935833, "cos_r_negg": [0.15263885259628296, 0.1746169924736023, 0.17576250433921814], "cos_apical_negg": [0.014830075204372406, 0.033648520708084106, 0.08620042353868484], "cos_Ac_negg": [0.1543373167514801, 0.17593152821063995, 0.17378994822502136], "r_norm": [0.14228959381580353, 0.1397356390953064, 0.13788512349128723]}, {"step": 5900, "epoch": 12, "train_loss": 0.0014230222441256046, "cos_r_negg": [0.15213468670845032, 0.17430678009986877, 0.1759105622768402], "cos_apical_negg": [0.012498117052018642, 0.03378203883767128, 0.08660750836133957], "cos_Ac_negg": [0.15374501049518585, 0.17561297118663788, 0.17391635477542877], "r_norm": [0.14283820986747742, 0.1403321474790573, 0.13854818046092987]}, {"step": 6000, "epoch": 12, "train_loss": 0.000244228751398623, "cos_r_negg": [0.146570086479187, 0.16816240549087524, 0.16942447423934937], "cos_apical_negg": [0.012282353825867176, 0.03434045985341072, 0.08635909855365753], "cos_Ac_negg": [0.14840777218341827, 0.1694069504737854, 0.16741929948329926], "r_norm": [0.13565529882907867, 0.13311327993869781, 0.1313837468624115], "ldr": {"dL_sdil": -0.006922908127307892, "dL_bp": -0.012113668024539948, "ratio": 0.571495612500146}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.973, "test_loss": 0.14551787910461425}, {"step": 6100, "epoch": 13, "train_loss": 0.0009321304969489574, "cos_r_negg": [0.1442718654870987, 0.16489380598068237, 0.16624633967876434], "cos_apical_negg": [0.01497738342732191, 0.0343003086745739, 0.08436550945043564], "cos_Ac_negg": [0.14581488072872162, 0.16608846187591553, 0.16430988907814026], "r_norm": [0.1405264437198639, 0.13788649439811707, 0.1359935700893402]}, {"step": 6200, "epoch": 13, "train_loss": 0.0005231219693087041, "cos_r_negg": [0.14243698120117188, 0.16336093842983246, 0.16461198031902313], "cos_apical_negg": [0.016248907893896103, 0.034383758902549744, 0.08057625591754913], "cos_Ac_negg": [0.1440506875514984, 0.16449984908103943, 0.16269151866436005], "r_norm": [0.14291684329509735, 0.14038163423538208, 0.13856256008148193]}, {"step": 6300, "epoch": 13, "train_loss": 0.001865783822722733, "cos_r_negg": [0.13952532410621643, 0.1600104421377182, 0.16166721284389496], "cos_apical_negg": [0.014771247282624245, 0.03334880247712135, 0.07756161689758301], "cos_Ac_negg": [0.14130479097366333, 0.1612267941236496, 0.15993809700012207], "r_norm": [0.13932643830776215, 0.13680164515972137, 0.13506843149662018]}, {"step": 6400, "epoch": 13, "train_loss": 0.0006112951086834073, "cos_r_negg": [0.14089155197143555, 0.16116607189178467, 0.16280370950698853], "cos_apical_negg": [0.015558218583464622, 0.03493750840425491, 0.07947418838739395], "cos_Ac_negg": [0.14259234070777893, 0.16207370162010193, 0.16091573238372803], "r_norm": [0.14438581466674805, 0.14188647270202637, 0.14009180665016174]}, {"step": 6500, "epoch": 13, "train_loss": 3.8397760363295674e-05, "cos_r_negg": [0.1383352279663086, 0.15873318910598755, 0.1603013277053833], "cos_apical_negg": [0.012088337913155556, 0.03465346246957779, 0.08074767887592316], "cos_Ac_negg": [0.13982851803302765, 0.15974904596805573, 0.15843066573143005], "r_norm": [0.13894405961036682, 0.136375293135643, 0.1346203088760376], "ldr": {"dL_sdil": -0.007670000195503235, "dL_bp": -0.013794049620628357, "ratio": 0.5560368714371672}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9728, "test_loss": 0.14837855529785157}, {"step": 6600, "epoch": 14, "train_loss": 0.00018488842761144042, "cos_r_negg": [0.1388585865497589, 0.15941071510314941, 0.16098912060260773], "cos_apical_negg": [0.01295116450637579, 0.03385521471500397, 0.07871631532907486], "cos_Ac_negg": [0.14056354761123657, 0.16052661836147308, 0.1592254787683487], "r_norm": [0.1403844654560089, 0.13781742751598358, 0.13589346408843994]}, {"step": 6700, "epoch": 14, "train_loss": 0.0006927501526661217, "cos_r_negg": [0.1378195881843567, 0.15828436613082886, 0.16061481833457947], "cos_apical_negg": [0.01167018711566925, 0.0341699942946434, 0.07943053543567657], "cos_Ac_negg": [0.1394639015197754, 0.1593516319990158, 0.15883086621761322], "r_norm": [0.1413549929857254, 0.13879427313804626, 0.13706524670124054]}, {"step": 6800, "epoch": 14, "train_loss": 0.0006496164714917541, "cos_r_negg": [0.13616202771663666, 0.1556755155324936, 0.15808901190757751], "cos_apical_negg": [0.01172585692256689, 0.03351667895913124, 0.07742087543010712], "cos_Ac_negg": [0.13787169754505157, 0.15690740942955017, 0.15643104910850525], "r_norm": [0.13826745748519897, 0.1356661468744278, 0.1338145136833191]}, {"step": 6900, "epoch": 14, "train_loss": 0.0005739419721066952, "cos_r_negg": [0.13428537547588348, 0.1538451462984085, 0.15598808228969574], "cos_apical_negg": [0.010659871622920036, 0.03287579491734505, 0.07809679210186005], "cos_Ac_negg": [0.13613955676555634, 0.1549883782863617, 0.1543464958667755], "r_norm": [0.1374746710062027, 0.134980708360672, 0.13311296701431274]}, {"step": 7000, "epoch": 14, "train_loss": 0.00041817521560005844, "cos_r_negg": [0.13292250037193298, 0.15273667871952057, 0.15462860465049744], "cos_apical_negg": [0.012884172610938549, 0.03437041491270065, 0.07802854478359222], "cos_Ac_negg": [0.1347857415676117, 0.15392033755779266, 0.15295986831188202], "r_norm": [0.14128488302230835, 0.13878695666790009, 0.13697242736816406], "ldr": {"dL_sdil": -0.007651582360267639, "dL_bp": -0.013698846101760864, "ratio": 0.5585567064136955}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9731, "test_loss": 0.15074674072265626}], "final": {"test_acc": 0.9731, "test_loss": 0.15074674072265626, "wall_s": 42.02177023887634, "cos_r_negg": [0.13281261920928955, 0.15254908800125122, 0.1545860320329666], "cos_apical_negg": [0.012150068767368793, 0.03370468318462372, 0.07878153026103973], "cos_Ac_negg": [0.1346888244152069, 0.15379570424556732, 0.15292029082775116]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho0_res0_s0.json b/results/diag_nuis_v2_rho0_res0_s0.json new file mode 100644 index 0000000..ea40584 --- /dev/null +++ b/results/diag_nuis_v2_rho0_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho0_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3978137969970703, "cos_r_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "cos_apical_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "cos_Ac_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "r_norm": [4.891855239868164, 4.819446563720703, 4.813877105712891], "ldr": {"dL_sdil": -0.007513999938964844, "dL_bp": -0.23199224472045898, "ratio": 0.03238901346904463}}, {"step": 100, "epoch": 0, "train_loss": 0.3326342701911926, "cos_r_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "cos_apical_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "cos_Ac_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "r_norm": [0.8364949822425842, 0.8173195123672485, 0.8178015947341919]}, {"step": 200, "epoch": 0, "train_loss": 0.2806997001171112, "cos_r_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "cos_apical_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "cos_Ac_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "r_norm": [0.7367554306983948, 0.7248662710189819, 0.7212702035903931]}, {"step": 300, "epoch": 0, "train_loss": 0.2950687110424042, "cos_r_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "cos_apical_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "cos_Ac_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "r_norm": [0.5894073843955994, 0.580111563205719, 0.5781813859939575]}, {"step": 400, "epoch": 0, "train_loss": 0.11995674669742584, "cos_r_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "cos_apical_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "cos_Ac_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "r_norm": [0.5368587374687195, 0.524689793586731, 0.5227817893028259]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9455, "test_loss": 0.183567440032959}, {"step": 500, "epoch": 1, "train_loss": 0.266681432723999, "cos_r_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "cos_apical_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "cos_Ac_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "r_norm": [0.5609025359153748, 0.5504555702209473, 0.5459269881248474], "ldr": {"dL_sdil": -0.01507239043712616, "dL_bp": -0.012114167213439941, "ratio": 1.2441953434821542}}, {"step": 600, "epoch": 1, "train_loss": 0.26136401295661926, "cos_r_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "cos_apical_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "cos_Ac_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "r_norm": [0.48265740275382996, 0.47054633498191833, 0.469404935836792]}, {"step": 700, "epoch": 1, "train_loss": 0.05521097034215927, "cos_r_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "cos_apical_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "cos_Ac_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "r_norm": [0.38553091883659363, 0.3786587417125702, 0.3761131763458252]}, {"step": 800, "epoch": 1, "train_loss": 0.1167590469121933, "cos_r_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "cos_apical_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "cos_Ac_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "r_norm": [0.39521896839141846, 0.39017635583877563, 0.38718611001968384]}, {"step": 900, "epoch": 1, "train_loss": 0.12190888077020645, "cos_r_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "cos_apical_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "cos_Ac_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "r_norm": [0.3485320806503296, 0.3397080898284912, 0.33907851576805115]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9563, "test_loss": 0.14598272552490235}, {"step": 1000, "epoch": 2, "train_loss": 0.21153098344802856, "cos_r_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "cos_apical_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "cos_Ac_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "r_norm": [0.370191752910614, 0.36559224128723145, 0.3626173436641693], "ldr": {"dL_sdil": -0.010546453297138214, "dL_bp": -0.00943966954946518, "ratio": 1.1172481453798073}}, {"step": 1100, "epoch": 2, "train_loss": 0.08847551047801971, "cos_r_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "cos_apical_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "cos_Ac_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "r_norm": [0.2634318768978119, 0.25687265396118164, 0.2574353814125061]}, {"step": 1200, "epoch": 2, "train_loss": 0.0304334107786417, "cos_r_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "cos_apical_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "cos_Ac_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "r_norm": [0.3200984597206116, 0.3153305649757385, 0.310947060585022]}, {"step": 1300, "epoch": 2, "train_loss": 0.1379236876964569, "cos_r_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "cos_apical_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "cos_Ac_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "r_norm": [0.2996469736099243, 0.2942494750022888, 0.2918258309364319]}, {"step": 1400, "epoch": 2, "train_loss": 0.2675240635871887, "cos_r_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "cos_apical_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "cos_Ac_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "r_norm": [0.3374136984348297, 0.3291166424751282, 0.3261111378669739]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.961, "test_loss": 0.1328551145553589}, {"step": 1500, "epoch": 3, "train_loss": 0.02847268432378769, "cos_r_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "cos_apical_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "cos_Ac_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "r_norm": [0.2855607867240906, 0.2791121304035187, 0.2787642478942871], "ldr": {"dL_sdil": -0.006588086485862732, "dL_bp": -0.006262242794036865, "ratio": 1.0520330658747608}}, {"step": 1600, "epoch": 3, "train_loss": 0.0700979083776474, "cos_r_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "cos_apical_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "cos_Ac_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "r_norm": [0.2842082977294922, 0.2784254848957062, 0.27576351165771484]}, {"step": 1700, "epoch": 3, "train_loss": 0.08875302970409393, "cos_r_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "cos_apical_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "cos_Ac_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "r_norm": [0.28283974528312683, 0.27689802646636963, 0.27501118183135986]}, {"step": 1800, "epoch": 3, "train_loss": 0.08456698805093765, "cos_r_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "cos_apical_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "cos_Ac_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "r_norm": [0.2775178849697113, 0.2699923515319824, 0.2691011428833008]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9654, "test_loss": 0.11376221809387208}, {"step": 1900, "epoch": 4, "train_loss": 0.06340675801038742, "cos_r_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "cos_apical_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "cos_Ac_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "r_norm": [0.2617102265357971, 0.25448280572891235, 0.25316107273101807]}, {"step": 2000, "epoch": 4, "train_loss": 0.09591986238956451, "cos_r_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "cos_apical_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "cos_Ac_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "r_norm": [0.25978973507881165, 0.25386324524879456, 0.25181859731674194], "ldr": {"dL_sdil": -0.0057485103607177734, "dL_bp": -0.0059058889746665955, "ratio": 0.9733522565994891}}, {"step": 2100, "epoch": 4, "train_loss": 0.06319307535886765, "cos_r_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "cos_apical_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "cos_Ac_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "r_norm": [0.23031632602214813, 0.22501714527606964, 0.22409947216510773]}, {"step": 2200, "epoch": 4, "train_loss": 0.057848647236824036, "cos_r_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "cos_apical_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "cos_Ac_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "r_norm": [0.20740358531475067, 0.20326529443264008, 0.2015407234430313]}, {"step": 2300, "epoch": 4, "train_loss": 0.11177002638578415, "cos_r_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "cos_apical_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "cos_Ac_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "r_norm": [0.2687836289405823, 0.2638903856277466, 0.2608131766319275]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9672, "test_loss": 0.11259154510498047}, {"step": 2400, "epoch": 5, "train_loss": 0.020450670272111893, "cos_r_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "cos_apical_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "cos_Ac_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "r_norm": [0.23890022933483124, 0.23350372910499573, 0.2330227941274643]}, {"step": 2500, "epoch": 5, "train_loss": 0.04949410259723663, "cos_r_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "cos_apical_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "cos_Ac_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "r_norm": [0.19714675843715668, 0.19294852018356323, 0.1915959119796753], "ldr": {"dL_sdil": -0.005470342934131622, "dL_bp": -0.0060305967926979065, "ratio": 0.9070981068996915}}, {"step": 2600, "epoch": 5, "train_loss": 0.02772347629070282, "cos_r_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "cos_apical_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "cos_Ac_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "r_norm": [0.21218553185462952, 0.2091660499572754, 0.20675276219844818]}, {"step": 2700, "epoch": 5, "train_loss": 0.09122030436992645, "cos_r_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "cos_apical_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "cos_Ac_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "r_norm": [0.20776993036270142, 0.20399165153503418, 0.20168794691562653]}, {"step": 2800, "epoch": 5, "train_loss": 0.011094631627202034, "cos_r_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "cos_apical_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "cos_Ac_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "r_norm": [0.1927286684513092, 0.18906784057617188, 0.18704429268836975]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9689, "test_loss": 0.11404035301208496}, {"step": 2900, "epoch": 6, "train_loss": 0.0365702323615551, "cos_r_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "cos_apical_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "cos_Ac_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "r_norm": [0.2347666174173355, 0.22903230786323547, 0.22849947214126587]}, {"step": 3000, "epoch": 6, "train_loss": 0.014387298375368118, "cos_r_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "cos_apical_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "cos_Ac_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "r_norm": [0.18399973213672638, 0.18022119998931885, 0.1777746081352234], "ldr": {"dL_sdil": -0.006564125418663025, "dL_bp": -0.007992811501026154, "ratio": 0.8212536249378948}}, {"step": 3100, "epoch": 6, "train_loss": 0.043194446712732315, "cos_r_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "cos_apical_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "cos_Ac_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "r_norm": [0.17794635891914368, 0.17384949326515198, 0.17293182015419006]}, {"step": 3200, "epoch": 6, "train_loss": 0.06848011910915375, "cos_r_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "cos_apical_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "cos_Ac_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "r_norm": [0.1944277584552765, 0.18918633460998535, 0.1876811534166336]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9699, "test_loss": 0.11801956939697265}, {"step": 3300, "epoch": 7, "train_loss": 0.01468692533671856, "cos_r_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "cos_apical_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "cos_Ac_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "r_norm": [0.15375901758670807, 0.151533305644989, 0.14959079027175903]}, {"step": 3400, "epoch": 7, "train_loss": 0.009123355150222778, "cos_r_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "cos_apical_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "cos_Ac_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "r_norm": [0.1616383194923401, 0.1594000905752182, 0.1570923626422882]}, {"step": 3500, "epoch": 7, "train_loss": 0.017021456733345985, "cos_r_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "cos_apical_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "cos_Ac_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "r_norm": [0.15971358120441437, 0.15718066692352295, 0.15513375401496887], "ldr": {"dL_sdil": -0.00407978892326355, "dL_bp": -0.0053077638149261475, "ratio": 0.768645528610492}}, {"step": 3600, "epoch": 7, "train_loss": 0.0063411155715584755, "cos_r_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "cos_apical_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "cos_Ac_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "r_norm": [0.15865351259708405, 0.15548795461654663, 0.15406224131584167]}, {"step": 3700, "epoch": 7, "train_loss": 0.005838679149746895, "cos_r_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "cos_apical_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "cos_Ac_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "r_norm": [0.17863257229328156, 0.17555025219917297, 0.1733328402042389]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.97, "test_loss": 0.12616020851135254}, {"step": 3800, "epoch": 8, "train_loss": 0.08425555378198624, "cos_r_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "cos_apical_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "cos_Ac_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "r_norm": [0.15988609194755554, 0.15718698501586914, 0.15486183762550354]}, {"step": 3900, "epoch": 8, "train_loss": 0.009738568216562271, "cos_r_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "cos_apical_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "cos_Ac_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "r_norm": [0.17696574330329895, 0.17444398999214172, 0.17152070999145508]}, {"step": 4000, "epoch": 8, "train_loss": 0.0004337254213169217, "cos_r_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "cos_apical_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "cos_Ac_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "r_norm": [0.173438161611557, 0.16961370408535004, 0.16862860321998596], "ldr": {"dL_sdil": -0.006462305784225464, "dL_bp": -0.009116381406784058, "ratio": 0.7088674218277513}}, {"step": 4100, "epoch": 8, "train_loss": 0.010536604560911655, "cos_r_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "cos_apical_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "cos_Ac_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "r_norm": [0.14982295036315918, 0.14678488671779633, 0.1456487774848938]}, {"step": 4200, "epoch": 8, "train_loss": 0.009496842510998249, "cos_r_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "cos_apical_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "cos_Ac_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "r_norm": [0.1480349898338318, 0.145409494638443, 0.14320231974124908]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9716, "test_loss": 0.12644243202209474}, {"step": 4300, "epoch": 9, "train_loss": 0.01001154538244009, "cos_r_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "cos_apical_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "cos_Ac_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "r_norm": [0.16093958914279938, 0.15813371539115906, 0.15620499849319458]}, {"step": 4400, "epoch": 9, "train_loss": 0.0019738487899303436, "cos_r_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "cos_apical_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "cos_Ac_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "r_norm": [0.15255683660507202, 0.1499708890914917, 0.1482204645872116]}, {"step": 4500, "epoch": 9, "train_loss": 0.0038066008128225803, "cos_r_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "cos_apical_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "cos_Ac_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "r_norm": [0.1395764797925949, 0.1376519650220871, 0.13562631607055664], "ldr": {"dL_sdil": -0.005997225642204285, "dL_bp": -0.008848287165164948, "ratio": 0.6777837936606441}}, {"step": 4600, "epoch": 9, "train_loss": 0.011752025224268436, "cos_r_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "cos_apical_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "cos_Ac_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "r_norm": [0.147353857755661, 0.14492204785346985, 0.14291857182979584]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9727, "test_loss": 0.12896337757110596}, {"step": 4700, "epoch": 10, "train_loss": 0.0019247026648372412, "cos_r_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "cos_apical_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "cos_Ac_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "r_norm": [0.13999506831169128, 0.13813723623752594, 0.1367495357990265]}, {"step": 4800, "epoch": 10, "train_loss": 0.0010178547818213701, "cos_r_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "cos_apical_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "cos_Ac_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "r_norm": [0.15259207785129547, 0.150788351893425, 0.14868605136871338]}, {"step": 4900, "epoch": 10, "train_loss": 0.001019246643409133, "cos_r_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "cos_apical_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "cos_Ac_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "r_norm": [0.14551062881946564, 0.14358925819396973, 0.14128731191158295]}, {"step": 5000, "epoch": 10, "train_loss": 0.0010785356862470508, "cos_r_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "cos_apical_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "cos_Ac_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "r_norm": [0.13078957796096802, 0.12892042100429535, 0.12751910090446472], "ldr": {"dL_sdil": -0.004710696637630463, "dL_bp": -0.007292650640010834, "ratio": 0.6459512281838122}}, {"step": 5100, "epoch": 10, "train_loss": 0.0011707834200933576, "cos_r_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "cos_apical_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "cos_Ac_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "r_norm": [0.13529300689697266, 0.133498877286911, 0.1317964643239975]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9726, "test_loss": 0.13503065013885499}, {"step": 5200, "epoch": 11, "train_loss": 0.0011789256241172552, "cos_r_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "cos_apical_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "cos_Ac_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "r_norm": [0.13868428766727448, 0.13704298436641693, 0.13511855900287628]}, {"step": 5300, "epoch": 11, "train_loss": 0.0012689412105828524, "cos_r_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "cos_apical_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "cos_Ac_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "r_norm": [0.1293502151966095, 0.12786594033241272, 0.12634214758872986]}, {"step": 5400, "epoch": 11, "train_loss": 0.0007522239466197789, "cos_r_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "cos_apical_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "cos_Ac_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "r_norm": [0.13768011331558228, 0.13615527749061584, 0.1340952068567276]}, {"step": 5500, "epoch": 11, "train_loss": 0.002217905130237341, "cos_r_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "cos_apical_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "cos_Ac_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "r_norm": [0.12865038216114044, 0.12704630196094513, 0.12513038516044617], "ldr": {"dL_sdil": -0.005985930562019348, "dL_bp": -0.009874477982521057, "ratio": 0.6062022288788452}}, {"step": 5600, "epoch": 11, "train_loss": 0.0008059403044171631, "cos_r_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "cos_apical_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "cos_Ac_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "r_norm": [0.1310221403837204, 0.12954476475715637, 0.12770068645477295]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9729, "test_loss": 0.14027503452301027}, {"step": 5700, "epoch": 12, "train_loss": 0.00024155571009032428, "cos_r_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "cos_apical_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "cos_Ac_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "r_norm": [0.13277700543403625, 0.13118377327919006, 0.12941528856754303]}, {"step": 5800, "epoch": 12, "train_loss": 0.00038036395562812686, "cos_r_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "cos_apical_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "cos_Ac_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "r_norm": [0.1338832974433899, 0.13205818831920624, 0.12990295886993408]}, {"step": 5900, "epoch": 12, "train_loss": 0.001817264361307025, "cos_r_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "cos_apical_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "cos_Ac_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "r_norm": [0.13398057222366333, 0.13230293989181519, 0.13060159981250763]}, {"step": 6000, "epoch": 12, "train_loss": 0.0019937404431402683, "cos_r_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "cos_apical_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "cos_Ac_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "r_norm": [0.1423790156841278, 0.14043937623500824, 0.1384449005126953], "ldr": {"dL_sdil": -0.0069742351770401, "dL_bp": -0.011977218091487885, "ratio": 0.582291741184594}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9732, "test_loss": 0.14210308151245118}, {"step": 6100, "epoch": 13, "train_loss": 0.0017001167871057987, "cos_r_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "cos_apical_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "cos_Ac_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "r_norm": [0.1407199203968048, 0.13882631063461304, 0.13691718876361847]}, {"step": 6200, "epoch": 13, "train_loss": 0.00037529002293013036, "cos_r_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "cos_apical_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "cos_Ac_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "r_norm": [0.13515688478946686, 0.13340508937835693, 0.13153940439224243]}, {"step": 6300, "epoch": 13, "train_loss": 0.00033519911812618375, "cos_r_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "cos_apical_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "cos_Ac_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "r_norm": [0.13734742999076843, 0.13553787767887115, 0.13386204838752747]}, {"step": 6400, "epoch": 13, "train_loss": 0.0003998060419689864, "cos_r_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "cos_apical_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "cos_Ac_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "r_norm": [0.13284258544445038, 0.1312873661518097, 0.12953701615333557]}, {"step": 6500, "epoch": 13, "train_loss": 0.00031235706410370767, "cos_r_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "cos_apical_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "cos_Ac_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "r_norm": [0.13349714875221252, 0.1318046897649765, 0.13015931844711304], "ldr": {"dL_sdil": -0.006666526198387146, "dL_bp": -0.011768236756324768, "ratio": 0.5664847110425665}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9736, "test_loss": 0.1460468635559082}, {"step": 6600, "epoch": 14, "train_loss": 0.0015140974428504705, "cos_r_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "cos_apical_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "cos_Ac_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "r_norm": [0.1308879405260086, 0.12910108268260956, 0.12746359407901764]}, {"step": 6700, "epoch": 14, "train_loss": 0.0013997508212924004, "cos_r_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "cos_apical_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "cos_Ac_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "r_norm": [0.13147181272506714, 0.12963689863681793, 0.1280984878540039]}, {"step": 6800, "epoch": 14, "train_loss": 0.0015433531953021884, "cos_r_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "cos_apical_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "cos_Ac_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "r_norm": [0.13564874231815338, 0.13395941257476807, 0.13222013413906097]}, {"step": 6900, "epoch": 14, "train_loss": 0.0010377462022006512, "cos_r_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "cos_apical_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "cos_Ac_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "r_norm": [0.13570146262645721, 0.13370797038078308, 0.1319391280412674]}, {"step": 7000, "epoch": 14, "train_loss": 0.00036846072180196643, "cos_r_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "cos_apical_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "cos_Ac_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "r_norm": [0.1375933289527893, 0.13558220863342285, 0.1339113861322403], "ldr": {"dL_sdil": -0.006832651793956757, "dL_bp": -0.012336961925029755, "ratio": 0.5538358499829995}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9731, "test_loss": 0.14842316989898682}], "final": {"test_acc": 0.9731, "test_loss": 0.14842316989898682, "wall_s": 37.175920248031616, "cos_r_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414], "cos_apical_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414], "cos_Ac_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho0_res1_s0.json b/results/diag_nuis_v2_rho0_res1_s0.json new file mode 100644 index 0000000..e474937 --- /dev/null +++ b/results/diag_nuis_v2_rho0_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho0_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3978137969970703, "cos_r_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "cos_apical_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "cos_Ac_negg": [-0.02239900827407837, -0.021294258534908295, 0.034575555473566055], "r_norm": [4.891855239868164, 4.819446563720703, 4.813877105712891], "ldr": {"dL_sdil": -0.007513999938964844, "dL_bp": -0.23199224472045898, "ratio": 0.03238901346904463}}, {"step": 100, "epoch": 0, "train_loss": 0.3326342701911926, "cos_r_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "cos_apical_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "cos_Ac_negg": [0.44732898473739624, 0.38416725397109985, 0.3667517900466919], "r_norm": [0.8364949822425842, 0.8173195123672485, 0.8178015947341919]}, {"step": 200, "epoch": 0, "train_loss": 0.2806997001171112, "cos_r_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "cos_apical_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "cos_Ac_negg": [0.47987207770347595, 0.4345510005950928, 0.43154287338256836], "r_norm": [0.7367554306983948, 0.7248662710189819, 0.7212702035903931]}, {"step": 300, "epoch": 0, "train_loss": 0.2950687110424042, "cos_r_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "cos_apical_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "cos_Ac_negg": [0.5101386308670044, 0.4740510582923889, 0.4546336531639099], "r_norm": [0.5894073843955994, 0.580111563205719, 0.5781813859939575]}, {"step": 400, "epoch": 0, "train_loss": 0.11995674669742584, "cos_r_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "cos_apical_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "cos_Ac_negg": [0.5342420339584351, 0.48684829473495483, 0.45452749729156494], "r_norm": [0.5368587374687195, 0.524689793586731, 0.5227817893028259]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9455, "test_loss": 0.183567440032959}, {"step": 500, "epoch": 1, "train_loss": 0.266681432723999, "cos_r_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "cos_apical_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "cos_Ac_negg": [0.5489188432693481, 0.508673906326294, 0.4944116771221161], "r_norm": [0.5609025359153748, 0.5504555702209473, 0.5459269881248474], "ldr": {"dL_sdil": -0.01507239043712616, "dL_bp": -0.012114167213439941, "ratio": 1.2441953434821542}}, {"step": 600, "epoch": 1, "train_loss": 0.26136401295661926, "cos_r_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "cos_apical_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "cos_Ac_negg": [0.5187579393386841, 0.5020226836204529, 0.4812536835670471], "r_norm": [0.48265740275382996, 0.47054633498191833, 0.469404935836792]}, {"step": 700, "epoch": 1, "train_loss": 0.05521097034215927, "cos_r_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "cos_apical_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "cos_Ac_negg": [0.529063880443573, 0.5166817903518677, 0.49424171447753906], "r_norm": [0.38553091883659363, 0.3786587417125702, 0.3761131763458252]}, {"step": 800, "epoch": 1, "train_loss": 0.1167590469121933, "cos_r_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "cos_apical_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "cos_Ac_negg": [0.5241111516952515, 0.5325952768325806, 0.48102712631225586], "r_norm": [0.39521896839141846, 0.39017635583877563, 0.38718611001968384]}, {"step": 900, "epoch": 1, "train_loss": 0.12190888077020645, "cos_r_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "cos_apical_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "cos_Ac_negg": [0.556609034538269, 0.5399625301361084, 0.5007081031799316], "r_norm": [0.3485320806503296, 0.3397080898284912, 0.33907851576805115]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9563, "test_loss": 0.14598272552490235}, {"step": 1000, "epoch": 2, "train_loss": 0.21153098344802856, "cos_r_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "cos_apical_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "cos_Ac_negg": [0.5525427460670471, 0.5603439211845398, 0.5186597108840942], "r_norm": [0.370191752910614, 0.36559224128723145, 0.3626173436641693], "ldr": {"dL_sdil": -0.010546453297138214, "dL_bp": -0.00943966954946518, "ratio": 1.1172481453798073}}, {"step": 1100, "epoch": 2, "train_loss": 0.08847551047801971, "cos_r_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "cos_apical_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "cos_Ac_negg": [0.5112615823745728, 0.5405528545379639, 0.491232693195343], "r_norm": [0.2634318768978119, 0.25687265396118164, 0.2574353814125061]}, {"step": 1200, "epoch": 2, "train_loss": 0.0304334107786417, "cos_r_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "cos_apical_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "cos_Ac_negg": [0.5318017601966858, 0.5454596877098083, 0.5005227327346802], "r_norm": [0.3200984597206116, 0.3153305649757385, 0.310947060585022]}, {"step": 1300, "epoch": 2, "train_loss": 0.1379236876964569, "cos_r_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "cos_apical_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "cos_Ac_negg": [0.5548704862594604, 0.5711873769760132, 0.5169695615768433], "r_norm": [0.2996469736099243, 0.2942494750022888, 0.2918258309364319]}, {"step": 1400, "epoch": 2, "train_loss": 0.2675240635871887, "cos_r_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "cos_apical_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "cos_Ac_negg": [0.5424305200576782, 0.5742589235305786, 0.49545636773109436], "r_norm": [0.3374136984348297, 0.3291166424751282, 0.3261111378669739]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.961, "test_loss": 0.1328551145553589}, {"step": 1500, "epoch": 3, "train_loss": 0.02847268432378769, "cos_r_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "cos_apical_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "cos_Ac_negg": [0.5073622465133667, 0.5457200407981873, 0.4972744882106781], "r_norm": [0.2855607867240906, 0.2791121304035187, 0.2787642478942871], "ldr": {"dL_sdil": -0.006588086485862732, "dL_bp": -0.006262242794036865, "ratio": 1.0520330658747608}}, {"step": 1600, "epoch": 3, "train_loss": 0.0700979083776474, "cos_r_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "cos_apical_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "cos_Ac_negg": [0.5162990093231201, 0.5577571988105774, 0.5150010585784912], "r_norm": [0.2842082977294922, 0.2784254848957062, 0.27576351165771484]}, {"step": 1700, "epoch": 3, "train_loss": 0.08875302970409393, "cos_r_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "cos_apical_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "cos_Ac_negg": [0.5350693464279175, 0.5757480263710022, 0.5141506791114807], "r_norm": [0.28283974528312683, 0.27689802646636963, 0.27501118183135986]}, {"step": 1800, "epoch": 3, "train_loss": 0.08456698805093765, "cos_r_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "cos_apical_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "cos_Ac_negg": [0.5278818011283875, 0.5696221590042114, 0.5028855204582214], "r_norm": [0.2775178849697113, 0.2699923515319824, 0.2691011428833008]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9654, "test_loss": 0.11376221809387208}, {"step": 1900, "epoch": 4, "train_loss": 0.06340675801038742, "cos_r_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "cos_apical_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "cos_Ac_negg": [0.5159375071525574, 0.5532490611076355, 0.4990085959434509], "r_norm": [0.2617102265357971, 0.25448280572891235, 0.25316107273101807]}, {"step": 2000, "epoch": 4, "train_loss": 0.09591986238956451, "cos_r_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "cos_apical_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "cos_Ac_negg": [0.47856640815734863, 0.5316683053970337, 0.4704130291938782], "r_norm": [0.25978973507881165, 0.25386324524879456, 0.25181859731674194], "ldr": {"dL_sdil": -0.0057485103607177734, "dL_bp": -0.0059058889746665955, "ratio": 0.9733522565994891}}, {"step": 2100, "epoch": 4, "train_loss": 0.06319307535886765, "cos_r_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "cos_apical_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "cos_Ac_negg": [0.45975053310394287, 0.5218613147735596, 0.4650382995605469], "r_norm": [0.23031632602214813, 0.22501714527606964, 0.22409947216510773]}, {"step": 2200, "epoch": 4, "train_loss": 0.057848647236824036, "cos_r_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "cos_apical_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "cos_Ac_negg": [0.479114830493927, 0.5228580236434937, 0.4676823019981384], "r_norm": [0.20740358531475067, 0.20326529443264008, 0.2015407234430313]}, {"step": 2300, "epoch": 4, "train_loss": 0.11177002638578415, "cos_r_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "cos_apical_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "cos_Ac_negg": [0.4605717957019806, 0.510822057723999, 0.4517856240272522], "r_norm": [0.2687836289405823, 0.2638903856277466, 0.2608131766319275]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9672, "test_loss": 0.11259154510498047}, {"step": 2400, "epoch": 5, "train_loss": 0.020450670272111893, "cos_r_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "cos_apical_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "cos_Ac_negg": [0.4683169722557068, 0.5224604606628418, 0.4606319069862366], "r_norm": [0.23890022933483124, 0.23350372910499573, 0.2330227941274643]}, {"step": 2500, "epoch": 5, "train_loss": 0.04949410259723663, "cos_r_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "cos_apical_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "cos_Ac_negg": [0.4616062641143799, 0.5154315233230591, 0.45055413246154785], "r_norm": [0.19714675843715668, 0.19294852018356323, 0.1915959119796753], "ldr": {"dL_sdil": -0.005470342934131622, "dL_bp": -0.0060305967926979065, "ratio": 0.9070981068996915}}, {"step": 2600, "epoch": 5, "train_loss": 0.02772347629070282, "cos_r_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "cos_apical_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "cos_Ac_negg": [0.450645387172699, 0.5067758560180664, 0.45540016889572144], "r_norm": [0.21218553185462952, 0.2091660499572754, 0.20675276219844818]}, {"step": 2700, "epoch": 5, "train_loss": 0.09122030436992645, "cos_r_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "cos_apical_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "cos_Ac_negg": [0.46868619322776794, 0.5180381536483765, 0.450004905462265], "r_norm": [0.20776993036270142, 0.20399165153503418, 0.20168794691562653]}, {"step": 2800, "epoch": 5, "train_loss": 0.011094631627202034, "cos_r_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "cos_apical_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "cos_Ac_negg": [0.39923107624053955, 0.45298054814338684, 0.41231513023376465], "r_norm": [0.1927286684513092, 0.18906784057617188, 0.18704429268836975]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9689, "test_loss": 0.11404035301208496}, {"step": 2900, "epoch": 6, "train_loss": 0.0365702323615551, "cos_r_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "cos_apical_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "cos_Ac_negg": [0.36084771156311035, 0.4027143120765686, 0.3576774001121521], "r_norm": [0.2347666174173355, 0.22903230786323547, 0.22849947214126587]}, {"step": 3000, "epoch": 6, "train_loss": 0.014387298375368118, "cos_r_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "cos_apical_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "cos_Ac_negg": [0.38353437185287476, 0.43034154176712036, 0.3947371542453766], "r_norm": [0.18399973213672638, 0.18022119998931885, 0.1777746081352234], "ldr": {"dL_sdil": -0.006564125418663025, "dL_bp": -0.007992811501026154, "ratio": 0.8212536249378948}}, {"step": 3100, "epoch": 6, "train_loss": 0.043194446712732315, "cos_r_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "cos_apical_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "cos_Ac_negg": [0.3321278393268585, 0.38589146733283997, 0.3507818579673767], "r_norm": [0.17794635891914368, 0.17384949326515198, 0.17293182015419006]}, {"step": 3200, "epoch": 6, "train_loss": 0.06848011910915375, "cos_r_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "cos_apical_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "cos_Ac_negg": [0.3206913471221924, 0.37333035469055176, 0.33710089325904846], "r_norm": [0.1944277584552765, 0.18918633460998535, 0.1876811534166336]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9699, "test_loss": 0.11801956939697265}, {"step": 3300, "epoch": 7, "train_loss": 0.01468692533671856, "cos_r_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "cos_apical_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "cos_Ac_negg": [0.3207220435142517, 0.3707606792449951, 0.33233171701431274], "r_norm": [0.15375901758670807, 0.151533305644989, 0.14959079027175903]}, {"step": 3400, "epoch": 7, "train_loss": 0.009123355150222778, "cos_r_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "cos_apical_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "cos_Ac_negg": [0.28374552726745605, 0.3274969458580017, 0.2981197237968445], "r_norm": [0.1616383194923401, 0.1594000905752182, 0.1570923626422882]}, {"step": 3500, "epoch": 7, "train_loss": 0.017021456733345985, "cos_r_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "cos_apical_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "cos_Ac_negg": [0.30124372243881226, 0.3384850025177002, 0.3080917298793793], "r_norm": [0.15971358120441437, 0.15718066692352295, 0.15513375401496887], "ldr": {"dL_sdil": -0.00407978892326355, "dL_bp": -0.0053077638149261475, "ratio": 0.768645528610492}}, {"step": 3600, "epoch": 7, "train_loss": 0.0063411155715584755, "cos_r_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "cos_apical_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "cos_Ac_negg": [0.30416423082351685, 0.34374314546585083, 0.30913037061691284], "r_norm": [0.15865351259708405, 0.15548795461654663, 0.15406224131584167]}, {"step": 3700, "epoch": 7, "train_loss": 0.005838679149746895, "cos_r_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "cos_apical_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "cos_Ac_negg": [0.27810975909233093, 0.32046079635620117, 0.29577505588531494], "r_norm": [0.17863257229328156, 0.17555025219917297, 0.1733328402042389]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.97, "test_loss": 0.12616020851135254}, {"step": 3800, "epoch": 8, "train_loss": 0.08425555378198624, "cos_r_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "cos_apical_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "cos_Ac_negg": [0.2577763795852661, 0.2929215431213379, 0.26875755190849304], "r_norm": [0.15988609194755554, 0.15718698501586914, 0.15486183762550354]}, {"step": 3900, "epoch": 8, "train_loss": 0.009738568216562271, "cos_r_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "cos_apical_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "cos_Ac_negg": [0.26248815655708313, 0.3003271818161011, 0.27528297901153564], "r_norm": [0.17696574330329895, 0.17444398999214172, 0.17152070999145508]}, {"step": 4000, "epoch": 8, "train_loss": 0.0004337254213169217, "cos_r_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "cos_apical_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "cos_Ac_negg": [0.23113428056240082, 0.26362621784210205, 0.24363160133361816], "r_norm": [0.173438161611557, 0.16961370408535004, 0.16862860321998596], "ldr": {"dL_sdil": -0.006462305784225464, "dL_bp": -0.009116381406784058, "ratio": 0.7088674218277513}}, {"step": 4100, "epoch": 8, "train_loss": 0.010536604560911655, "cos_r_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "cos_apical_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "cos_Ac_negg": [0.23122937977313995, 0.26008719205856323, 0.24139222502708435], "r_norm": [0.14982295036315918, 0.14678488671779633, 0.1456487774848938]}, {"step": 4200, "epoch": 8, "train_loss": 0.009496842510998249, "cos_r_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "cos_apical_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "cos_Ac_negg": [0.2148902714252472, 0.24816416203975677, 0.22951406240463257], "r_norm": [0.1480349898338318, 0.145409494638443, 0.14320231974124908]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9716, "test_loss": 0.12644243202209474}, {"step": 4300, "epoch": 9, "train_loss": 0.01001154538244009, "cos_r_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "cos_apical_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "cos_Ac_negg": [0.21183210611343384, 0.24454396963119507, 0.227471262216568], "r_norm": [0.16093958914279938, 0.15813371539115906, 0.15620499849319458]}, {"step": 4400, "epoch": 9, "train_loss": 0.0019738487899303436, "cos_r_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "cos_apical_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "cos_Ac_negg": [0.19909998774528503, 0.229609876871109, 0.2141648679971695], "r_norm": [0.15255683660507202, 0.1499708890914917, 0.1482204645872116]}, {"step": 4500, "epoch": 9, "train_loss": 0.0038066008128225803, "cos_r_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "cos_apical_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "cos_Ac_negg": [0.18125811219215393, 0.21069471538066864, 0.1968364417552948], "r_norm": [0.1395764797925949, 0.1376519650220871, 0.13562631607055664], "ldr": {"dL_sdil": -0.005997225642204285, "dL_bp": -0.008848287165164948, "ratio": 0.6777837936606441}}, {"step": 4600, "epoch": 9, "train_loss": 0.011752025224268436, "cos_r_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "cos_apical_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "cos_Ac_negg": [0.19314654171466827, 0.22306716442108154, 0.21022829413414001], "r_norm": [0.147353857755661, 0.14492204785346985, 0.14291857182979584]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9727, "test_loss": 0.12896337757110596}, {"step": 4700, "epoch": 10, "train_loss": 0.0019247026648372412, "cos_r_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "cos_apical_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "cos_Ac_negg": [0.20044180750846863, 0.2280893176794052, 0.2148800492286682], "r_norm": [0.13999506831169128, 0.13813723623752594, 0.1367495357990265]}, {"step": 4800, "epoch": 10, "train_loss": 0.0010178547818213701, "cos_r_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "cos_apical_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "cos_Ac_negg": [0.18150821328163147, 0.20863491296768188, 0.19851279258728027], "r_norm": [0.15259207785129547, 0.150788351893425, 0.14868605136871338]}, {"step": 4900, "epoch": 10, "train_loss": 0.001019246643409133, "cos_r_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "cos_apical_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "cos_Ac_negg": [0.1805490404367447, 0.2097999006509781, 0.19830119609832764], "r_norm": [0.14551062881946564, 0.14358925819396973, 0.14128731191158295]}, {"step": 5000, "epoch": 10, "train_loss": 0.0010785356862470508, "cos_r_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "cos_apical_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "cos_Ac_negg": [0.1676468402147293, 0.19102786481380463, 0.18200671672821045], "r_norm": [0.13078957796096802, 0.12892042100429535, 0.12751910090446472], "ldr": {"dL_sdil": -0.004710696637630463, "dL_bp": -0.007292650640010834, "ratio": 0.6459512281838122}}, {"step": 5100, "epoch": 10, "train_loss": 0.0011707834200933576, "cos_r_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "cos_apical_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "cos_Ac_negg": [0.16800034046173096, 0.1939515322446823, 0.18445813655853271], "r_norm": [0.13529300689697266, 0.133498877286911, 0.1317964643239975]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9726, "test_loss": 0.13503065013885499}, {"step": 5200, "epoch": 11, "train_loss": 0.0011789256241172552, "cos_r_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "cos_apical_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "cos_Ac_negg": [0.1572558730840683, 0.18001487851142883, 0.17088842391967773], "r_norm": [0.13868428766727448, 0.13704298436641693, 0.13511855900287628]}, {"step": 5300, "epoch": 11, "train_loss": 0.0012689412105828524, "cos_r_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "cos_apical_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "cos_Ac_negg": [0.16087812185287476, 0.18432626128196716, 0.17572149634361267], "r_norm": [0.1293502151966095, 0.12786594033241272, 0.12634214758872986]}, {"step": 5400, "epoch": 11, "train_loss": 0.0007522239466197789, "cos_r_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "cos_apical_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "cos_Ac_negg": [0.15411823987960815, 0.17842479050159454, 0.16960319876670837], "r_norm": [0.13768011331558228, 0.13615527749061584, 0.1340952068567276]}, {"step": 5500, "epoch": 11, "train_loss": 0.002217905130237341, "cos_r_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "cos_apical_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "cos_Ac_negg": [0.15328368544578552, 0.17673790454864502, 0.16894632577896118], "r_norm": [0.12865038216114044, 0.12704630196094513, 0.12513038516044617], "ldr": {"dL_sdil": -0.005985930562019348, "dL_bp": -0.009874477982521057, "ratio": 0.6062022288788452}}, {"step": 5600, "epoch": 11, "train_loss": 0.0008059403044171631, "cos_r_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "cos_apical_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "cos_Ac_negg": [0.14735251665115356, 0.1706986427307129, 0.16230052709579468], "r_norm": [0.1310221403837204, 0.12954476475715637, 0.12770068645477295]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9729, "test_loss": 0.14027503452301027}, {"step": 5700, "epoch": 12, "train_loss": 0.00024155571009032428, "cos_r_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "cos_apical_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "cos_Ac_negg": [0.14655159413814545, 0.17031948268413544, 0.16324445605278015], "r_norm": [0.13277700543403625, 0.13118377327919006, 0.12941528856754303]}, {"step": 5800, "epoch": 12, "train_loss": 0.00038036395562812686, "cos_r_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "cos_apical_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "cos_Ac_negg": [0.1454523503780365, 0.16926605999469757, 0.16230905055999756], "r_norm": [0.1338832974433899, 0.13205818831920624, 0.12990295886993408]}, {"step": 5900, "epoch": 12, "train_loss": 0.001817264361307025, "cos_r_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "cos_apical_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "cos_Ac_negg": [0.14318090677261353, 0.165847510099411, 0.15835271775722504], "r_norm": [0.13398057222366333, 0.13230293989181519, 0.13060159981250763]}, {"step": 6000, "epoch": 12, "train_loss": 0.0019937404431402683, "cos_r_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "cos_apical_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "cos_Ac_negg": [0.14001017808914185, 0.16120645403862, 0.1555197834968567], "r_norm": [0.1423790156841278, 0.14043937623500824, 0.1384449005126953], "ldr": {"dL_sdil": -0.0069742351770401, "dL_bp": -0.011977218091487885, "ratio": 0.582291741184594}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9732, "test_loss": 0.14210308151245118}, {"step": 6100, "epoch": 13, "train_loss": 0.0017001167871057987, "cos_r_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "cos_apical_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "cos_Ac_negg": [0.1384393870830536, 0.15989845991134644, 0.15438809990882874], "r_norm": [0.1407199203968048, 0.13882631063461304, 0.13691718876361847]}, {"step": 6200, "epoch": 13, "train_loss": 0.00037529002293013036, "cos_r_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "cos_apical_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "cos_Ac_negg": [0.13541299104690552, 0.1572113186120987, 0.1518983691930771], "r_norm": [0.13515688478946686, 0.13340508937835693, 0.13153940439224243]}, {"step": 6300, "epoch": 13, "train_loss": 0.00033519911812618375, "cos_r_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "cos_apical_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "cos_Ac_negg": [0.13752731680870056, 0.15933454036712646, 0.1538372039794922], "r_norm": [0.13734742999076843, 0.13553787767887115, 0.13386204838752747]}, {"step": 6400, "epoch": 13, "train_loss": 0.0003998060419689864, "cos_r_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "cos_apical_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "cos_Ac_negg": [0.13656961917877197, 0.15909065306186676, 0.1524784117937088], "r_norm": [0.13284258544445038, 0.1312873661518097, 0.12953701615333557]}, {"step": 6500, "epoch": 13, "train_loss": 0.00031235706410370767, "cos_r_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "cos_apical_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "cos_Ac_negg": [0.13381841778755188, 0.15428520739078522, 0.14882761240005493], "r_norm": [0.13349714875221252, 0.1318046897649765, 0.13015931844711304], "ldr": {"dL_sdil": -0.006666526198387146, "dL_bp": -0.011768236756324768, "ratio": 0.5664847110425665}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9736, "test_loss": 0.1460468635559082}, {"step": 6600, "epoch": 14, "train_loss": 0.0015140974428504705, "cos_r_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "cos_apical_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "cos_Ac_negg": [0.13230784237384796, 0.15244686603546143, 0.14781415462493896], "r_norm": [0.1308879405260086, 0.12910108268260956, 0.12746359407901764]}, {"step": 6700, "epoch": 14, "train_loss": 0.0013997508212924004, "cos_r_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "cos_apical_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "cos_Ac_negg": [0.12985539436340332, 0.14975348114967346, 0.14552007615566254], "r_norm": [0.13147181272506714, 0.12963689863681793, 0.1280984878540039]}, {"step": 6800, "epoch": 14, "train_loss": 0.0015433531953021884, "cos_r_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "cos_apical_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "cos_Ac_negg": [0.12913398444652557, 0.14907878637313843, 0.14442747831344604], "r_norm": [0.13564874231815338, 0.13395941257476807, 0.13222013413906097]}, {"step": 6900, "epoch": 14, "train_loss": 0.0010377462022006512, "cos_r_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "cos_apical_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "cos_Ac_negg": [0.12653705477714539, 0.14591452479362488, 0.14195674657821655], "r_norm": [0.13570146262645721, 0.13370797038078308, 0.1319391280412674]}, {"step": 7000, "epoch": 14, "train_loss": 0.00036846072180196643, "cos_r_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "cos_apical_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "cos_Ac_negg": [0.12514552474021912, 0.14422184228897095, 0.13999950885772705], "r_norm": [0.1375933289527893, 0.13558220863342285, 0.1339113861322403], "ldr": {"dL_sdil": -0.006832651793956757, "dL_bp": -0.012336961925029755, "ratio": 0.5538358499829995}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9731, "test_loss": 0.14842316989898682}], "final": {"test_acc": 0.9731, "test_loss": 0.14842316989898682, "wall_s": 36.81630539894104, "cos_r_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414], "cos_apical_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414], "cos_Ac_negg": [0.12640921771526337, 0.1457367241382599, 0.141432985663414]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho0p5_res0_s0.json b/results/diag_nuis_v2_rho0p5_res0_s0.json new file mode 100644 index 0000000..e0174be --- /dev/null +++ b/results/diag_nuis_v2_rho0p5_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.5, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho0p5_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.011882033199071884, -0.02912256121635437, 0.03196502849459648], "cos_apical_negg": [-0.011126603931188583, -0.02555844560265541, 0.021645020693540573], "cos_Ac_negg": [-0.011882638558745384, -0.029123816639184952, 0.03196432814002037], "r_norm": [4.8995513916015625, 4.827376365661621, 4.823241233825684], "ldr": {"dL_sdil": 0.06155204772949219, "dL_bp": -0.18675637245178223, "ratio": -0.3295847253907442}}, {"step": 100, "epoch": 0, "train_loss": 11.01133918762207, "cos_r_negg": [-0.01831037551164627, -0.062168240547180176, 0.0016092935111373663], "cos_apical_negg": [-0.024052806198596954, -0.050919994711875916, 0.002923077205196023], "cos_Ac_negg": [-0.018309377133846283, -0.06216748058795929, 0.001609420171007514], "r_norm": [5.75061559677124, 5.754624843597412, 5.879786968231201]}, {"step": 200, "epoch": 0, "train_loss": 16.102767944335938, "cos_r_negg": [-0.011801040731370449, -0.037153564393520355, 0.0003752322809305042], "cos_apical_negg": [-0.007682319730520248, -0.023321431130170822, -0.003912506625056267], "cos_Ac_negg": [-0.011801381595432758, -0.03715430200099945, 0.0003748911549337208], "r_norm": [5.4998393058776855, 5.5523834228515625, 5.428306579589844]}, {"step": 300, "epoch": 0, "train_loss": 12.778050422668457, "cos_r_negg": [0.0011825775727629662, -0.008718792349100113, 1.9686409359565005e-05], "cos_apical_negg": [0.0016850177198648453, -0.014440877363085747, 0.0008777290349826217], "cos_Ac_negg": [0.0011833556927740574, -0.008722281083464622, 1.9742508811759762e-05], "r_norm": [4.418936252593994, 4.686548233032227, 4.742585182189941]}, {"step": 400, "epoch": 0, "train_loss": 9.024974822998047, "cos_r_negg": [-0.005862962454557419, -0.030997376888990402, -0.00020035452325828373], "cos_apical_negg": [-0.002142856828868389, -0.017748422920703888, -0.002730628475546837], "cos_Ac_negg": [-0.005862319841980934, -0.03099626675248146, -0.00020048883743584156], "r_norm": [4.0902791023254395, 4.433443069458008, 4.404144287109375]}, {"epoch_end": 0, "step": 469, "test_acc": 0.1032, "test_loss": 16.48041630859375}, {"step": 500, "epoch": 1, "train_loss": 13.55573844909668, "cos_r_negg": [0.0018942637834697962, -0.06467853486537933, 0.00012749381130561233], "cos_apical_negg": [-0.006424483377486467, -0.019106460735201836, -0.0018795760115608573], "cos_Ac_negg": [0.0018968733493238688, -0.06467785686254501, 0.00012740657257381827], "r_norm": [3.632378578186035, 4.037422180175781, 4.037497520446777], "ldr": {"dL_sdil": -1.660703182220459, "dL_bp": -1.660703182220459, "ratio": 1.0}}, {"step": 600, "epoch": 1, "train_loss": 10.38748550415039, "cos_r_negg": [0.007109517231583595, -0.08064009249210358, -0.00011612967500695959], "cos_apical_negg": [0.0028138221241533756, -0.039357807487249374, -0.00372366514056921], "cos_Ac_negg": [0.007110614329576492, -0.08064048737287521, -0.00011625832121353596], "r_norm": [3.301206588745117, 3.8002684116363525, 3.799511432647705]}, {"step": 700, "epoch": 1, "train_loss": 16.875041961669922, "cos_r_negg": [-0.013645504601299763, -0.05567697063088417, 0.00023865894763730466], "cos_apical_negg": [-0.009853346273303032, -0.04029817506670952, -0.0018670851131901145], "cos_Ac_negg": [-0.013644423335790634, -0.05567646026611328, 0.0002385975676588714], "r_norm": [2.998892068862915, 3.543501377105713, 3.5417983531951904]}, {"step": 800, "epoch": 1, "train_loss": 11.963838577270508, "cos_r_negg": [0.0192817822098732, 0.00446967501193285, -0.0011514746583998203], "cos_apical_negg": [0.0025333655066788197, 0.03951650857925415, -0.0025745639577507973], "cos_Ac_negg": [0.01928340457379818, 0.004468689206987619, -0.0011515086516737938], "r_norm": [2.454526901245117, 3.0481646060943604, 3.0456655025482178]}, {"step": 900, "epoch": 1, "train_loss": 11.593137741088867, "cos_r_negg": [-0.002850255463272333, 0.014964073896408081, -0.0006848533521406353], "cos_apical_negg": [-0.00502749253064394, 0.017335057258605957, -0.0024890012573450804], "cos_Ac_negg": [-0.002849691081792116, 0.014963455498218536, -0.0006848884513601661], "r_norm": [1.9261387586593628, 2.4771203994750977, 2.606520414352417]}, {"epoch_end": 1, "step": 938, "test_acc": 0.0974, "test_loss": 13.94229423828125}, {"step": 1000, "epoch": 2, "train_loss": 16.2428035736084, "cos_r_negg": [0.021700456738471985, -0.005125678144395351, -0.0002760242496151477], "cos_apical_negg": [0.009458078071475029, -0.029055830091238022, -0.0008412611205130816], "cos_Ac_negg": [0.021700531244277954, -0.0051255496218800545, -0.0002760332718025893], "r_norm": [1.9479341506958008, 2.5511507987976074, 2.677489757537842], "ldr": {"dL_sdil": -0.99554443359375, "dL_bp": -0.99554443359375, "ratio": 1.0}}, {"step": 1100, "epoch": 2, "train_loss": 11.126084327697754, "cos_r_negg": [-0.02017492800951004, -0.02059192955493927, 0.00044770236127078533], "cos_apical_negg": [-0.012671062722802162, 0.007235139608383179, -0.000937682983931154], "cos_Ac_negg": [-0.020174665376544, -0.02059197425842285, 0.00044768431689590216], "r_norm": [1.8259186744689941, 2.430492877960205, 2.4384374618530273]}, {"step": 1200, "epoch": 2, "train_loss": 12.048250198364258, "cos_r_negg": [-0.01335977017879486, 0.0023609984200447798, -0.00012080677697667852], "cos_apical_negg": [-0.011809613555669785, 0.017779191955924034, -0.000480127171613276], "cos_Ac_negg": [-0.013360636308789253, 0.0023595287930220366, -0.00012081054592272267], "r_norm": [1.8168751001358032, 2.4779646396636963, 2.506977081298828]}, {"step": 1300, "epoch": 2, "train_loss": 17.05799102783203, "cos_r_negg": [-0.0023031141608953476, -0.02995016612112522, -8.517334936186671e-05], "cos_apical_negg": [-0.000995587557554245, 0.018713703379034996, -0.0006097162258811295], "cos_Ac_negg": [-0.0023040329106152058, -0.0299530029296875, -8.51778604555875e-05], "r_norm": [1.7100799083709717, 2.3238143920898438, 2.420278549194336]}, {"step": 1400, "epoch": 2, "train_loss": 9.156953811645508, "cos_r_negg": [-0.002274569123983383, 0.01330319233238697, 0.00028448697412386537], "cos_apical_negg": [-0.007129235193133354, 0.008041895925998688, -0.0004620736581273377], "cos_Ac_negg": [-0.0022744094021618366, 0.013301555067300797, 0.00028448167722672224], "r_norm": [1.415165662765503, 1.9009230136871338, 1.973520040512085]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.0982, "test_loss": 8.7781197265625}, {"step": 1500, "epoch": 3, "train_loss": 15.432962417602539, "cos_r_negg": [-0.008353911340236664, 0.003334975801408291, -0.00028969504637643695], "cos_apical_negg": [0.001486289082095027, -0.004706281237304211, -0.00116568966768682], "cos_Ac_negg": [-0.00835498422384262, 0.0033329082652926445, -0.00028970016865059733], "r_norm": [1.269024133682251, 1.63621187210083, 1.8308550119400024], "ldr": {"dL_sdil": -1.1959495544433594, "dL_bp": -1.1959495544433594, "ratio": 1.0}}, {"step": 1600, "epoch": 3, "train_loss": 8.989423751831055, "cos_r_negg": [0.0005351777654141188, 0.0007288283668458462, 8.72880918905139e-05], "cos_apical_negg": [0.0003771310730371624, 0.0006364201544784009, -0.0003166344831697643], "cos_Ac_negg": [0.0005352185689844191, 0.0007288765627890825, 8.72861419338733e-05], "r_norm": [1.250170350074768, 1.6352180242538452, 1.7524869441986084]}, {"step": 1700, "epoch": 3, "train_loss": 9.063461303710938, "cos_r_negg": [-0.0023876880295574665, 0.0028360183350741863, -2.3778238755767234e-05], "cos_apical_negg": [0.0007244194275699556, -0.0015344558050855994, -2.5236537112505175e-05], "cos_Ac_negg": [-0.002387924352660775, 0.0028359713032841682, -2.377823147980962e-05], "r_norm": [1.0773367881774902, 1.417598009109497, 1.5776009559631348]}, {"step": 1800, "epoch": 3, "train_loss": 12.365030288696289, "cos_r_negg": [0.0004447380779311061, -0.00011181503214174882, -1.611530751688406e-05], "cos_apical_negg": [-4.741392331197858e-05, 0.0022714652586728334, -0.00012778016389347613], "cos_Ac_negg": [0.0004449016705621034, -0.00011169760546181351, -1.6115653124870732e-05], "r_norm": [1.08027982711792, 1.4740252494812012, 1.642350196838379]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.101, "test_loss": 7.567432421875}, {"step": 1900, "epoch": 4, "train_loss": 11.618885040283203, "cos_r_negg": [1.2476308256736957e-05, -2.1039646469489526e-07, 0.00024595187278464437], "cos_apical_negg": [-2.6646473997971043e-05, 1.2412034777753433e-07, -0.00029832933796569705], "cos_Ac_negg": [1.2478170901886187e-05, -2.1039932107669301e-07, 0.00024595041759312153], "r_norm": [1.1735512018203735, 1.504148006439209, 1.6478557586669922]}, {"step": 2000, "epoch": 4, "train_loss": 9.665514945983887, "cos_r_negg": [4.2355300138297025e-06, 1.61810786991623e-09, 1.6449826944153756e-05], "cos_apical_negg": [1.2519739129857044e-06, 1.0854774856738914e-08, -0.00016245104779955], "cos_Ac_negg": [4.235828782839235e-06, 1.618066347575109e-09, 1.6449455870315433e-05], "r_norm": [1.0755054950714111, 1.33332359790802, 1.4733424186706543], "ldr": {"dL_sdil": -2.6690845489501953, "dL_bp": -2.6690855026245117, "ratio": 0.9999996426962285}}, {"step": 2100, "epoch": 4, "train_loss": 11.35377311706543, "cos_r_negg": [5.732657655244111e-07, -5.307922504305118e-11, 6.716619100188836e-05], "cos_apical_negg": [9.716786735225469e-07, -1.4519410118651876e-09, -0.00017920631216838956], "cos_Ac_negg": [5.732431418437045e-07, -5.3081219975048555e-11, 6.716577627230436e-05], "r_norm": [1.076040506362915, 1.1840384006500244, 1.4105955362319946]}, {"step": 2200, "epoch": 4, "train_loss": 10.267271995544434, "cos_r_negg": [1.2288154493944603e-07, -2.8495195786293337e-11, 9.27407654671697e-06], "cos_apical_negg": [7.659499345891163e-08, -2.533125445514095e-11, 0.00019440380856394768], "cos_Ac_negg": [1.2288391815218347e-07, -2.8495331094724463e-11, 9.274352123611607e-06], "r_norm": [1.0076576471328735, 1.1821677684783936, 1.4083125591278076]}, {"step": 2300, "epoch": 4, "train_loss": 5.025246620178223, "cos_r_negg": [8.755226588164078e-08, -7.350250616489085e-12, 4.265413008397445e-05], "cos_apical_negg": [1.5032142641757673e-07, -4.163810407575852e-10, -0.00021684428793378174], "cos_Ac_negg": [8.754479097206058e-08, -7.35088639264303e-12, 4.265383176971227e-05], "r_norm": [0.8376622200012207, 0.9656543731689453, 1.168035864830017]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.1135, "test_loss": 7.6074916015625}, {"step": 2400, "epoch": 5, "train_loss": 13.367894172668457, "cos_r_negg": [1.4791265812164056e-07, -7.363101794943816e-11, 9.639024938223884e-06], "cos_apical_negg": [1.4987985252901126e-07, -6.703615440528665e-10, -0.0002883507404476404], "cos_Ac_negg": [1.4790870750402973e-07, -7.363178122776759e-11, 9.638738447392825e-06], "r_norm": [0.7853376865386963, 0.8800057768821716, 1.1286193132400513]}, {"step": 2500, "epoch": 5, "train_loss": 11.597400665283203, "cos_r_negg": [2.5269146775031004e-08, -1.0159933658271392e-11, 5.21527836099267e-05], "cos_apical_negg": [7.503008703224623e-08, -3.798339975880083e-10, -0.00012048051576130092], "cos_Ac_negg": [2.5266070124985163e-08, -1.0160327440500438e-11, 5.2152652642689645e-05], "r_norm": [0.7613652944564819, 0.8633002042770386, 1.1105130910873413], "ldr": {"dL_sdil": -1.531510353088379, "dL_bp": -1.531510353088379, "ratio": 1.0}}, {"step": 2600, "epoch": 5, "train_loss": 11.138182640075684, "cos_r_negg": [2.6174191702921235e-09, -4.9861049317145856e-11, -6.247340934351087e-05], "cos_apical_negg": [7.727852846528549e-08, -9.265282985282397e-10, -0.000657822412904352], "cos_Ac_negg": [2.6112338957773318e-09, -4.9861795248240526e-11, -6.247381679713726e-05], "r_norm": [0.8762261867523193, 0.9776666760444641, 1.393789291381836]}, {"step": 2700, "epoch": 5, "train_loss": 14.02318000793457, "cos_r_negg": [2.191944226126452e-08, -2.5405597764427412e-11, 0.00012618984328582883], "cos_apical_negg": [3.3974142610304625e-08, -6.201916757930803e-10, -0.00024270149879157543], "cos_Ac_negg": [2.1918628689832076e-08, -2.5406019302232075e-11, 0.00012618965411093086], "r_norm": [0.7268966436386108, 0.7734205722808838, 1.0983965396881104]}, {"step": 2800, "epoch": 5, "train_loss": 21.555164337158203, "cos_r_negg": [1.0135548933476457e-07, -8.922049210546845e-11, 1.1707807061611675e-05], "cos_apical_negg": [1.9724694766409812e-07, -8.00412514045945e-10, -0.0003447060298640281], "cos_Ac_negg": [1.0135568118130323e-07, -8.922090843910269e-11, 1.1707668818417005e-05], "r_norm": [0.7220097184181213, 0.7415224313735962, 1.0832834243774414]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.1032, "test_loss": 13.8519380859375}, {"step": 2900, "epoch": 6, "train_loss": 12.916088104248047, "cos_r_negg": [-3.952176363952731e-09, -2.6425937826868307e-11, -1.618498208699748e-05], "cos_apical_negg": [1.555431872191093e-08, -3.0461727584807363e-10, -0.00031154463067650795], "cos_Ac_negg": [-3.950803684205084e-09, -2.6426055788064673e-11, -1.6185093045351095e-05], "r_norm": [0.5892959237098694, 0.6278326511383057, 0.972356915473938]}, {"step": 3000, "epoch": 6, "train_loss": 10.466683387756348, "cos_r_negg": [1.4554649041542689e-08, 7.758914496135139e-13, 4.739173891721293e-05], "cos_apical_negg": [6.169445043724409e-08, -3.095968481581224e-10, -0.0002251363475807011], "cos_Ac_negg": [1.4553308780307361e-08, 7.757812404626807e-13, 4.739165888167918e-05], "r_norm": [0.5805792808532715, 0.6775977611541748, 1.0184144973754883], "ldr": {"dL_sdil": -1.724860668182373, "dL_bp": -1.724860668182373, "ratio": 1.0}}, {"step": 3100, "epoch": 6, "train_loss": 11.238972663879395, "cos_r_negg": [-4.847019674514286e-09, -3.3195141514036353e-12, 1.4812907465966418e-05], "cos_apical_negg": [-8.827498021446445e-08, -1.0334505473608147e-10, 6.654742173850536e-05], "cos_Ac_negg": [-4.844093126621374e-09, -3.3195562184479277e-12, 1.4812919289397541e-05], "r_norm": [0.5864731073379517, 0.7075850367546082, 1.0492979288101196]}, {"step": 3200, "epoch": 6, "train_loss": 7.564970016479492, "cos_r_negg": [1.4209851073587743e-09, -1.8516330829720928e-11, 4.6353206926141866e-06], "cos_apical_negg": [5.787335766171964e-08, -3.6114936041720114e-10, -0.00025268859462812543], "cos_Ac_negg": [1.4190835173621963e-09, -1.8516417565894727e-11, 4.635270670405589e-06], "r_norm": [0.6048710942268372, 0.6631734371185303, 1.0108771324157715]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.1032, "test_loss": 10.5336509765625}, {"step": 3300, "epoch": 7, "train_loss": 14.027457237243652, "cos_r_negg": [-8.153517860876036e-10, 1.3625140946049719e-11, 4.658469151763711e-06], "cos_apical_negg": [2.6563782284938497e-08, -2.920659825100813e-10, -0.00012199459888506681], "cos_Ac_negg": [-8.17524492546795e-10, 1.3625075026557631e-11, 4.658444595406763e-06], "r_norm": [0.6412185430526733, 0.7500307559967041, 1.124558448791504]}, {"step": 3400, "epoch": 7, "train_loss": 7.86048698425293, "cos_r_negg": [1.135503868709975e-08, -6.33874116390265e-12, 3.0985596822574735e-05], "cos_apical_negg": [2.1395225147102792e-08, -2.443308888544493e-10, -0.0001948322169482708], "cos_Ac_negg": [1.135296301413291e-08, -6.338779327819122e-12, 3.09855786326807e-05], "r_norm": [0.5041470527648926, 0.5709538459777832, 0.9465043544769287]}, {"step": 3500, "epoch": 7, "train_loss": 10.57800006866455, "cos_r_negg": [4.146490706347095e-09, -5.919634574191868e-12, 1.1776619430747814e-05], "cos_apical_negg": [2.9064224804642436e-08, -1.0355773183423622e-10, -0.00010957917402265593], "cos_Ac_negg": [4.145578103020853e-09, -5.9196475846179375e-12, 1.1776604878832586e-05], "r_norm": [0.48656338453292847, 0.5178446769714355, 0.9521645307540894], "ldr": {"dL_sdil": -3.427248954772949, "dL_bp": -3.427248954772949, "ratio": 1.0}}, {"step": 3600, "epoch": 7, "train_loss": 15.532432556152344, "cos_r_negg": [-2.8825335007809372e-08, -3.402247927830615e-11, 2.450506144668907e-06], "cos_apical_negg": [1.0475099898599183e-08, -4.666115271945159e-10, -0.00037971953861415386], "cos_Ac_negg": [-2.8828875286990296e-08, -3.4022538258904333e-11, 2.450475449222722e-06], "r_norm": [0.7184155583381653, 0.7601858377456665, 1.2098398208618164]}, {"step": 3700, "epoch": 7, "train_loss": 7.748431205749512, "cos_r_negg": [-1.0919315451829448e-09, -3.96488397669259e-12, 9.782284905668348e-06], "cos_apical_negg": [1.6719159390277127e-08, -9.32635392525416e-12, -7.3591859290900175e-06], "cos_Ac_negg": [-1.0922833748594485e-09, -3.96488397669259e-12, 9.782283086678945e-06], "r_norm": [0.33216363191604614, 0.36118388175964355, 0.7546311616897583]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.1032, "test_loss": 10.11491142578125}, {"step": 3800, "epoch": 8, "train_loss": 13.716079711914062, "cos_r_negg": [-2.525082809512469e-09, -1.667197629950934e-11, -7.844359060982242e-06], "cos_apical_negg": [3.388213087873737e-08, -2.9722344030425063e-10, -0.00015424194862134755], "cos_Ac_negg": [-2.5274617954096357e-09, -1.667199711619105e-11, -7.84437361289747e-06], "r_norm": [0.5023097395896912, 0.5091642141342163, 0.9516236186027527]}, {"step": 3900, "epoch": 8, "train_loss": 9.766668319702148, "cos_r_negg": [1.1431598778699481e-08, 6.8066715458448e-12, 1.551962486701086e-05], "cos_apical_negg": [-2.1652407866668e-08, -1.4061461717229662e-10, -4.16592629335355e-05], "cos_Ac_negg": [1.142718986102409e-08, 6.80666287222742e-12, 1.5519626686000265e-05], "r_norm": [0.43007969856262207, 0.45048585534095764, 0.8641888499259949]}, {"step": 4000, "epoch": 8, "train_loss": 12.247589111328125, "cos_r_negg": [-3.455334685042999e-09, -1.155519690348994e-12, 2.484763172105886e-05], "cos_apical_negg": [4.8499202875973424e-08, -1.5151646604039115e-10, -0.00014251109678298235], "cos_Ac_negg": [-3.4537386284227978e-09, -1.1555299902696325e-12, 2.484763172105886e-05], "r_norm": [0.3594617247581482, 0.38788747787475586, 0.8590507507324219], "ldr": {"dL_sdil": -1.3875160217285156, "dL_bp": -1.3875160217285156, "ratio": 1.0}}, {"step": 4100, "epoch": 8, "train_loss": 8.769721984863281, "cos_r_negg": [-1.8066941276373427e-09, -7.248824804295673e-12, -4.73234820219659e-07], "cos_apical_negg": [-1.7092649073902066e-08, -1.8484512531724562e-10, -7.829488458810374e-05], "cos_Ac_negg": [-1.8101276033632985e-09, -7.2488326105513146e-12, -4.73234820219659e-07], "r_norm": [0.4758274555206299, 0.4902147352695465, 0.9862357378005981]}, {"step": 4200, "epoch": 8, "train_loss": 12.886968612670898, "cos_r_negg": [-9.115757038102856e-09, -7.222104858595202e-12, 8.253476153186057e-06], "cos_apical_negg": [-1.0084664836540469e-07, -1.4621293065175678e-10, -0.00015677290502935648], "cos_Ac_negg": [-9.122219424284594e-09, -7.22210572595694e-12, 8.253469786723144e-06], "r_norm": [0.4356619119644165, 0.46199288964271545, 0.9345585107803345]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.1032, "test_loss": 8.94383095703125}, {"step": 4300, "epoch": 9, "train_loss": 18.384628295898438, "cos_r_negg": [2.1353995904860312e-08, 5.409813222390092e-13, 2.0792986106243916e-05], "cos_apical_negg": [-1.4468396614120138e-07, -1.3982057699146733e-12, 9.554717689752579e-05], "cos_Ac_negg": [2.134482457449849e-08, 5.409815390794437e-13, 2.079298792523332e-05], "r_norm": [0.38187021017074585, 0.3877905607223511, 0.8918505311012268]}, {"step": 4400, "epoch": 9, "train_loss": 5.2729716300964355, "cos_r_negg": [2.498618378865558e-08, 2.1929025435796223e-12, 9.40924473979976e-06], "cos_apical_negg": [-5.771231670337329e-08, -5.625934440534053e-11, -4.310788881412009e-06], "cos_Ac_negg": [2.4978591639523984e-08, 2.1929029772604913e-12, 9.409243830305059e-06], "r_norm": [0.3855942189693451, 0.3867957592010498, 0.7870667576789856]}, {"step": 4500, "epoch": 9, "train_loss": 12.622437477111816, "cos_r_negg": [-4.888492721732973e-09, 3.717502000677442e-12, 2.535005842219107e-05], "cos_apical_negg": [2.6132745745144348e-08, -1.6110911216227208e-10, -0.00014179563731886446], "cos_Ac_negg": [-4.887177329493397e-09, 3.71750286803918e-12, 2.5350056603201665e-05], "r_norm": [0.3789624273777008, 0.3840675354003906, 0.8639934062957764], "ldr": {"dL_sdil": -2.1758832931518555, "dL_bp": -2.1758832931518555, "ratio": 1.0}}, {"step": 4600, "epoch": 9, "train_loss": 10.693771362304688, "cos_r_negg": [1.2233137169914698e-08, -6.026890141466734e-13, 7.357426511589438e-06], "cos_apical_negg": [-1.3117193020661944e-07, -1.2064170495929005e-10, -5.489778413902968e-05], "cos_Ac_negg": [1.2221770262499376e-08, -6.026891225668907e-13, 7.3574274210841395e-06], "r_norm": [0.36265474557876587, 0.3674789071083069, 0.8984209299087524]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.0982, "test_loss": 8.6538541015625}, {"step": 4700, "epoch": 10, "train_loss": 15.58753490447998, "cos_r_negg": [-1.6622867526905338e-09, -1.1247917527934526e-11, 1.4649567674496211e-05], "cos_apical_negg": [3.014616822838434e-08, -1.8309123661630622e-10, -0.00021861086133867502], "cos_Ac_negg": [-1.66329017226019e-09, -1.1247917527934526e-11, 1.4649569493485615e-05], "r_norm": [0.4678865671157837, 0.4572031497955322, 0.9481086730957031]}, {"step": 4800, "epoch": 10, "train_loss": 11.759526252746582, "cos_r_negg": [-3.814489168973978e-09, 1.204425014583732e-12, 1.639542642806191e-05], "cos_apical_negg": [-2.884018499571539e-07, -5.11666889413398e-11, 5.462760600494221e-05], "cos_Ac_negg": [-3.8376488653568686e-09, 1.204424147221994e-12, 1.639542642806191e-05], "r_norm": [0.36603865027427673, 0.370371013879776, 0.886262059211731]}, {"step": 4900, "epoch": 10, "train_loss": 12.935401916503906, "cos_r_negg": [1.1451318115973663e-08, 1.243709475684529e-11, 4.384463682072237e-05], "cos_apical_negg": [2.421143108222168e-08, -1.859647991153679e-10, -0.0001346292847301811], "cos_Ac_negg": [1.1448257453139377e-08, 1.243709475684529e-11, 4.384464045870118e-05], "r_norm": [0.5512735247612, 0.5365611910820007, 1.0067330598831177]}, {"step": 5000, "epoch": 10, "train_loss": 11.624526977539062, "cos_r_negg": [-2.121429476531489e-09, -2.777215523525056e-12, 2.3229858925333247e-05], "cos_apical_negg": [1.0330499122801484e-08, -1.4943735138217562e-10, -0.00016368820797652006], "cos_Ac_negg": [-2.125075448944358e-09, -2.777216390886794e-12, 2.322986074432265e-05], "r_norm": [0.39658790826797485, 0.4046897292137146, 0.9666480422019958], "ldr": {"dL_sdil": -4.388715744018555, "dL_bp": -4.388715744018555, "ratio": 1.0}}, {"step": 5100, "epoch": 10, "train_loss": 8.720470428466797, "cos_r_negg": [3.857199004642098e-08, -3.561757225961637e-12, -1.023932668431371e-06], "cos_apical_negg": [1.3631539275138493e-07, -5.952170700762593e-11, -7.972757157403976e-05], "cos_Ac_negg": [3.8582875561132823e-08, -3.5617570091212025e-12, -1.0239319863103447e-06], "r_norm": [0.6275029182434082, 0.6465598344802856, 1.1351993083953857]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.1028, "test_loss": 15.24673896484375}, {"step": 5200, "epoch": 11, "train_loss": 10.840343475341797, "cos_r_negg": [-1.2931762327639262e-09, -2.3084307902709877e-12, 2.0102874259464443e-05], "cos_apical_negg": [6.155039500299608e-08, -1.13622139474856e-10, -0.00013829351519234478], "cos_Ac_negg": [-1.2935628124211007e-09, -2.3084305734305532e-12, 2.010287244047504e-05], "r_norm": [0.3784415125846863, 0.38472315669059753, 0.9812639355659485]}, {"step": 5300, "epoch": 11, "train_loss": 8.186874389648438, "cos_r_negg": [1.152460571418601e-09, -1.8962923679227695e-13, 1.6198308003367856e-05], "cos_apical_negg": [3.927280722848536e-09, -6.867956897638194e-11, -8.57274208101444e-05], "cos_Ac_negg": [1.1512895081722263e-09, -1.8962910126700538e-13, 1.6198308003367856e-05], "r_norm": [0.3641085624694824, 0.3648543059825897, 0.8487534523010254]}, {"step": 5400, "epoch": 11, "train_loss": 10.055963516235352, "cos_r_negg": [9.430340952576444e-10, 1.0400551947453707e-12, 7.606373401358724e-06], "cos_apical_negg": [2.170450486005393e-08, -3.563471046019728e-11, -3.9937844121595845e-05], "cos_Ac_negg": [9.42558031624685e-10, 1.040054869484719e-12, 7.606374310853425e-06], "r_norm": [0.49187690019607544, 0.48382246494293213, 0.9072948098182678]}, {"step": 5500, "epoch": 11, "train_loss": 14.818922996520996, "cos_r_negg": [4.9122721890171306e-08, -1.165534248975808e-11, 3.101564516327926e-07], "cos_apical_negg": [6.695618708363327e-07, -1.4040868467901646e-10, -0.0002047528250841424], "cos_Ac_negg": [4.913884765755938e-08, -1.1655341622396342e-11, 3.101588958998036e-07], "r_norm": [0.7507508993148804, 0.7378922700881958, 1.271790862083435], "ldr": {"dL_sdil": -7.702569961547852, "dL_bp": -7.702569961547852, "ratio": 1.0}}, {"step": 5600, "epoch": 11, "train_loss": 9.57944393157959, "cos_r_negg": [7.78763276088057e-09, -4.0772721028418946e-13, 2.0506548025878146e-05], "cos_apical_negg": [-7.28280440398521e-08, -8.992458860879182e-11, -0.00011066096340073273], "cos_Ac_negg": [7.782887223584112e-09, -4.0772777949033e-13, 2.0506548025878146e-05], "r_norm": [0.45555323362350464, 0.45631271600723267, 1.00935959815979]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.0958, "test_loss": 10.998844140625}, {"step": 5700, "epoch": 12, "train_loss": 7.412187576293945, "cos_r_negg": [1.5035448441835797e-08, 5.498130596637285e-12, 1.004432124318555e-05], "cos_apical_negg": [-4.687224475219409e-08, -4.314311755071465e-11, -2.496761953807436e-05], "cos_Ac_negg": [1.5033325695412714e-08, 5.498130596637285e-12, 1.0044319424196146e-05], "r_norm": [0.7272461652755737, 0.7217174768447876, 1.1298494338989258]}, {"step": 5800, "epoch": 12, "train_loss": 9.098715782165527, "cos_r_negg": [1.5585621682134843e-08, 2.7771597955333904e-12, 7.2759030445013195e-06], "cos_apical_negg": [-3.6572732398099106e-08, -2.6526735669163415e-11, -5.32453486812301e-06], "cos_Ac_negg": [1.5582948265091545e-08, 2.7771597955333904e-12, 7.275903953996021e-06], "r_norm": [0.526990532875061, 0.5239669680595398, 0.9069192409515381]}, {"step": 5900, "epoch": 12, "train_loss": 9.841585159301758, "cos_r_negg": [2.0750629659005426e-08, 2.99176720723604e-12, 5.269820576359052e-06], "cos_apical_negg": [-6.8765302785323e-08, -4.448593576844573e-11, -2.0002784367534332e-05], "cos_Ac_negg": [2.074650140571066e-08, 2.991767640916909e-12, 5.269820576359052e-06], "r_norm": [0.4710872769355774, 0.4667884409427643, 0.9132919907569885]}, {"step": 6000, "epoch": 12, "train_loss": 6.885409355163574, "cos_r_negg": [7.772415600015847e-09, 2.3257337895821184e-12, 1.2293621693970636e-05], "cos_apical_negg": [-6.267450558539167e-09, -3.976535073446641e-11, -4.9737227527657524e-05], "cos_Ac_negg": [7.770671217599556e-09, 2.3257333559012494e-12, 1.2293622603465337e-05], "r_norm": [0.47403645515441895, 0.4687521457672119, 0.8888131976127625], "ldr": {"dL_sdil": -2.2825021743774414, "dL_bp": -2.2825021743774414, "ratio": 1.0}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.098, "test_loss": 11.20027314453125}, {"step": 6100, "epoch": 13, "train_loss": 10.357048034667969, "cos_r_negg": [-5.5274833599128215e-09, -3.899935929752019e-12, 1.8961232854053378e-05], "cos_apical_negg": [-3.7064545210796496e-08, -9.886789997803902e-11, -0.00012794848589692265], "cos_Ac_negg": [-5.531364255517701e-09, -3.89993549607115e-12, 1.8961232854053378e-05], "r_norm": [0.3995760679244995, 0.39526647329330444, 1.021580457687378]}, {"step": 6200, "epoch": 13, "train_loss": 18.197879791259766, "cos_r_negg": [8.840041587632186e-09, 1.9532370512664876e-12, 3.747525397557183e-06], "cos_apical_negg": [-6.497378279846089e-08, -3.88910501025741e-11, -1.76265984919155e-05], "cos_Ac_negg": [8.836033238424079e-09, 1.9532374849473566e-12, 3.747524715436157e-06], "r_norm": [0.2870451807975769, 0.28531473875045776, 0.7663108110427856]}, {"step": 6300, "epoch": 13, "train_loss": 11.638656616210938, "cos_r_negg": [1.6143841818916371e-09, -8.696009407352379e-13, 2.0303437850088812e-05], "cos_apical_negg": [-1.2088509748764409e-08, -1.1300485547316441e-10, -0.00013724472955800593], "cos_Ac_negg": [1.6090704324511762e-09, -8.696009407352379e-13, 2.0303439669078216e-05], "r_norm": [0.3190278112888336, 0.3195810914039612, 0.8861619830131531]}, {"step": 6400, "epoch": 13, "train_loss": 13.545377731323242, "cos_r_negg": [-5.226167942851134e-09, -4.288965883836315e-12, 1.1043461199733429e-05], "cos_apical_negg": [-1.786585102081517e-08, -9.10582997892817e-11, -0.0001209635374834761], "cos_Ac_negg": [-5.230543997924997e-09, -4.288965883836315e-12, 1.1043463928217534e-05], "r_norm": [0.49205517768859863, 0.49115192890167236, 1.0188000202178955]}, {"step": 6500, "epoch": 13, "train_loss": 7.112216949462891, "cos_r_negg": [-1.498677448807939e-08, 5.082673431439089e-12, 1.955018160515465e-05], "cos_apical_negg": [7.806637825069629e-08, -5.733822791564869e-11, -6.723613478243351e-05], "cos_Ac_negg": [-1.4982298068844102e-08, 5.082673865119958e-12, 1.955018160515465e-05], "r_norm": [0.5492075085639954, 0.542016863822937, 0.99283367395401], "ldr": {"dL_sdil": -1.6647114753723145, "dL_bp": -1.6647114753723145, "ratio": 1.0}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.1009, "test_loss": 13.61623701171875}, {"step": 6600, "epoch": 14, "train_loss": 5.0409650802612305, "cos_r_negg": [9.524525612647494e-10, 9.057832608960759e-13, 1.2884740499430336e-05], "cos_apical_negg": [-1.0824227558714483e-07, -6.094987015092812e-11, -5.930045881541446e-05], "cos_Ac_negg": [9.474647733043184e-10, 9.057830440556414e-13, 1.2884741408925038e-05], "r_norm": [0.43874382972717285, 0.4363207519054413, 0.8924903869628906]}, {"step": 6700, "epoch": 14, "train_loss": 19.058780670166016, "cos_r_negg": [-1.8265831513986086e-08, -4.201461227537617e-12, 1.2507634892244823e-05], "cos_apical_negg": [-2.8154056508356007e-07, -6.796535556574668e-11, 3.058017682633363e-05], "cos_Ac_negg": [-1.827444151558666e-08, -4.201460793856748e-12, 1.2507634892244823e-05], "r_norm": [0.7770832777023315, 0.775188684463501, 1.1878373622894287]}, {"step": 6800, "epoch": 14, "train_loss": 8.224281311035156, "cos_r_negg": [6.551137410326646e-09, 2.17405021936401e-12, 6.813340405642521e-06], "cos_apical_negg": [-1.4551302385257259e-08, -3.3061366144782056e-11, -2.206773206125945e-05], "cos_Ac_negg": [6.547717923410801e-09, 2.1740500025235754e-12, 6.813341315137222e-06], "r_norm": [0.3658082187175751, 0.3635425567626953, 0.846403181552887]}, {"step": 6900, "epoch": 14, "train_loss": 16.24734878540039, "cos_r_negg": [-3.023360228837646e-09, 8.597933129350643e-12, 3.163607470924035e-05], "cos_apical_negg": [1.5146621024086926e-07, -8.163396347793395e-11, -0.00010514531459193677], "cos_Ac_negg": [-3.0223672453644213e-09, 8.597933129350643e-12, 3.163607470924035e-05], "r_norm": [0.66852867603302, 0.6635032892227173, 1.0655066967010498]}, {"step": 7000, "epoch": 14, "train_loss": 11.85394287109375, "cos_r_negg": [3.534208259381444e-09, 2.7765708569132963e-12, 2.5216651920345612e-05], "cos_apical_negg": [-9.648308818555051e-09, 1.7697222159940296e-11, 8.077954407781363e-05], "cos_Ac_negg": [3.5342333504218004e-09, 2.7765708569132963e-12, 2.5216653739335015e-05], "r_norm": [1.1490089893341064, 1.1492254734039307, 1.4599595069885254], "ldr": {"dL_sdil": -1.3692293167114258, "dL_bp": -1.3692293167114258, "ratio": 1.0}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.0892, "test_loss": 9.79625048828125}], "final": {"test_acc": 0.0892, "test_loss": 9.79625048828125, "wall_s": 39.354275941848755, "cos_r_negg": [1.093324364376258e-08, 3.1049188827653174e-12, 7.1931413003767375e-06], "cos_apical_negg": [4.565207323281584e-09, -3.206007334610739e-11, 1.4505526451102924e-05], "cos_Ac_negg": [1.0927163174301313e-08, 3.1049193164461864e-12, 7.193141755124088e-06]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho0p5_res1_s0.json b/results/diag_nuis_v2_rho0p5_res1_s0.json new file mode 100644 index 0000000..a3bfe95 --- /dev/null +++ b/results/diag_nuis_v2_rho0p5_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.5, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho0p5_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.01951833814382553, -0.02552487701177597, 0.0337664932012558], "cos_apical_negg": [-0.01678706519305706, -0.02366996370255947, 0.024410059675574303], "cos_Ac_negg": [-0.019517866894602776, -0.025524867698550224, 0.03376641124486923], "r_norm": [4.898201942443848, 4.826807975769043, 4.823859214782715], "ldr": {"dL_sdil": -0.00920724868774414, "dL_bp": -0.24071121215820312, "ratio": 0.03825018620941031}}, {"step": 100, "epoch": 0, "train_loss": 0.20854580402374268, "cos_r_negg": [0.4560292363166809, 0.3767118453979492, 0.35909503698349], "cos_apical_negg": [0.08013659715652466, 0.08184289932250977, 0.13731014728546143], "cos_Ac_negg": [0.4591827690601349, 0.3799489736557007, 0.3613731861114502], "r_norm": [0.8228429555892944, 0.8063802123069763, 0.8042128682136536]}, {"step": 200, "epoch": 0, "train_loss": 0.33769410848617554, "cos_r_negg": [0.4648585915565491, 0.39329442381858826, 0.3825173079967499], "cos_apical_negg": [0.0687609612941742, 0.07838074862957001, 0.13243278861045837], "cos_Ac_negg": [0.47176802158355713, 0.4003522992134094, 0.3905133008956909], "r_norm": [0.739084005355835, 0.7225308418273926, 0.721724271774292]}, {"step": 300, "epoch": 0, "train_loss": 0.1765526682138443, "cos_r_negg": [0.48277807235717773, 0.4553801119327545, 0.42237886786460876], "cos_apical_negg": [0.05212590843439102, 0.06264340877532959, 0.11667770147323608], "cos_Ac_negg": [0.4867022633552551, 0.4572802186012268, 0.42461252212524414], "r_norm": [0.5614144206047058, 0.5511546730995178, 0.5513275861740112]}, {"step": 400, "epoch": 0, "train_loss": 0.16667333245277405, "cos_r_negg": [0.5168399810791016, 0.4739108085632324, 0.45380741357803345], "cos_apical_negg": [0.059294700622558594, 0.0732555165886879, 0.11268673837184906], "cos_Ac_negg": [0.520999550819397, 0.4751344919204712, 0.4541701078414917], "r_norm": [0.5608276128768921, 0.5486057996749878, 0.5492345690727234]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9469, "test_loss": 0.18007668762207032}, {"step": 500, "epoch": 1, "train_loss": 0.21246269345283508, "cos_r_negg": [0.5015144944190979, 0.48756474256515503, 0.4681017994880676], "cos_apical_negg": [0.0476343110203743, 0.06709837168455124, 0.12663201987743378], "cos_Ac_negg": [0.5121444463729858, 0.4922686219215393, 0.4720652997493744], "r_norm": [0.5095798969268799, 0.4929932951927185, 0.49390584230422974], "ldr": {"dL_sdil": -0.013827577233314514, "dL_bp": -0.01230853796005249, "ratio": 1.1234134613056468}}, {"step": 600, "epoch": 1, "train_loss": 0.05918657034635544, "cos_r_negg": [0.4865701496601105, 0.4759397506713867, 0.45569223165512085], "cos_apical_negg": [0.03844216465950012, 0.05055811256170273, 0.12032327055931091], "cos_Ac_negg": [0.50315922498703, 0.4858556389808655, 0.4645897150039673], "r_norm": [0.4355936646461487, 0.42486897110939026, 0.4232804477214813]}, {"step": 700, "epoch": 1, "train_loss": 0.20410266518592834, "cos_r_negg": [0.4908044934272766, 0.5028591156005859, 0.48066291213035583], "cos_apical_negg": [0.04429718106985092, 0.06336262822151184, 0.12432675063610077], "cos_Ac_negg": [0.5063460469245911, 0.5087595582008362, 0.48559415340423584], "r_norm": [0.4751412272453308, 0.46164628863334656, 0.46189165115356445]}, {"step": 800, "epoch": 1, "train_loss": 0.13557346165180206, "cos_r_negg": [0.5185573101043701, 0.5330528020858765, 0.5003015995025635], "cos_apical_negg": [0.029080187901854515, 0.05663395673036575, 0.11572425067424774], "cos_Ac_negg": [0.5284100770950317, 0.537632167339325, 0.5036655068397522], "r_norm": [0.3565206825733185, 0.3479672372341156, 0.34663116931915283]}, {"step": 900, "epoch": 1, "train_loss": 0.13805948197841644, "cos_r_negg": [0.5547139048576355, 0.535489559173584, 0.4738655090332031], "cos_apical_negg": [0.045282792299985886, 0.050790946930646896, 0.11138907074928284], "cos_Ac_negg": [0.5579003095626831, 0.5366984009742737, 0.4738525152206421], "r_norm": [0.4055035710334778, 0.3984900414943695, 0.39632460474967957]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9502, "test_loss": 0.17152556419372558}, {"step": 1000, "epoch": 2, "train_loss": 0.03167913854122162, "cos_r_negg": [0.5400651693344116, 0.5347380042076111, 0.48338890075683594], "cos_apical_negg": [0.03372028470039368, 0.06522026658058167, 0.1094057559967041], "cos_Ac_negg": [0.545549750328064, 0.5363824367523193, 0.4835861921310425], "r_norm": [0.33360567688941956, 0.32454732060432434, 0.32541853189468384], "ldr": {"dL_sdil": -0.007104687392711639, "dL_bp": -0.006297491490840912, "ratio": 1.128177370790372}}, {"step": 1100, "epoch": 2, "train_loss": 0.10970061272382736, "cos_r_negg": [0.5275467038154602, 0.5417709350585938, 0.5052412748336792], "cos_apical_negg": [0.03025328554213047, 0.06281273812055588, 0.11817236244678497], "cos_Ac_negg": [0.5325840711593628, 0.5437651872634888, 0.5041697025299072], "r_norm": [0.36377769708633423, 0.35069072246551514, 0.3537273705005646]}, {"step": 1200, "epoch": 2, "train_loss": 0.13575583696365356, "cos_r_negg": [0.5361267328262329, 0.551878809928894, 0.513104259967804], "cos_apical_negg": [0.03827253729104996, 0.047483813017606735, 0.10841576009988785], "cos_Ac_negg": [0.5398600101470947, 0.5541732311248779, 0.5119310021400452], "r_norm": [0.32667380571365356, 0.31916144490242004, 0.3197486698627472]}, {"step": 1300, "epoch": 2, "train_loss": 0.11548513174057007, "cos_r_negg": [0.5352425575256348, 0.5446561574935913, 0.5101741552352905], "cos_apical_negg": [0.02243058569729328, 0.04999538138508797, 0.11733421683311462], "cos_Ac_negg": [0.5404654741287231, 0.5476483106613159, 0.5061925053596497], "r_norm": [0.282317578792572, 0.27592933177948, 0.27469491958618164]}, {"step": 1400, "epoch": 2, "train_loss": 0.12009767442941666, "cos_r_negg": [0.4991346597671509, 0.5178653597831726, 0.4899614155292511], "cos_apical_negg": [0.030357830226421356, 0.06180844455957413, 0.11568541824817657], "cos_Ac_negg": [0.5048224329948425, 0.5198115110397339, 0.484002947807312], "r_norm": [0.3103398382663727, 0.2993900775909424, 0.29845768213272095]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9645, "test_loss": 0.12571081123352051}, {"step": 1500, "epoch": 3, "train_loss": 0.07781251519918442, "cos_r_negg": [0.5123809576034546, 0.5280567407608032, 0.49606454372406006], "cos_apical_negg": [0.02144334465265274, 0.061054203659296036, 0.1217404305934906], "cos_Ac_negg": [0.5203201770782471, 0.5322085618972778, 0.4916454553604126], "r_norm": [0.277731329202652, 0.26940208673477173, 0.269745409488678], "ldr": {"dL_sdil": -0.006865143775939941, "dL_bp": -0.006735637784004211, "ratio": 1.01922698281717}}, {"step": 1600, "epoch": 3, "train_loss": 0.0836792141199112, "cos_r_negg": [0.5033053159713745, 0.5245782732963562, 0.5115085244178772], "cos_apical_negg": [0.022051731124520302, 0.05579838529229164, 0.1165999323129654], "cos_Ac_negg": [0.5085608959197998, 0.5287084579467773, 0.5073779821395874], "r_norm": [0.28448835015296936, 0.27350759506225586, 0.27482765913009644]}, {"step": 1700, "epoch": 3, "train_loss": 0.08251354098320007, "cos_r_negg": [0.4871635437011719, 0.5151824951171875, 0.4837074875831604], "cos_apical_negg": [0.015474541112780571, 0.06020599603652954, 0.10684400796890259], "cos_Ac_negg": [0.48993363976478577, 0.5166673064231873, 0.4813007712364197], "r_norm": [0.27673575282096863, 0.2675589323043823, 0.2678774297237396]}, {"step": 1800, "epoch": 3, "train_loss": 0.03554634377360344, "cos_r_negg": [0.4917357563972473, 0.5261809825897217, 0.502294659614563], "cos_apical_negg": [0.022976141422986984, 0.05727025121450424, 0.11616137623786926], "cos_Ac_negg": [0.4978327751159668, 0.5316082239151001, 0.4992503225803375], "r_norm": [0.2829076647758484, 0.2745828628540039, 0.2731527090072632]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9644, "test_loss": 0.12461709251403809}, {"step": 1900, "epoch": 4, "train_loss": 0.03338209539651871, "cos_r_negg": [0.49845021963119507, 0.5365836024284363, 0.5142247676849365], "cos_apical_negg": [0.023470964282751083, 0.05410866066813469, 0.12844881415367126], "cos_Ac_negg": [0.5034118890762329, 0.539482057094574, 0.5103837251663208], "r_norm": [0.2652546465396881, 0.257359117269516, 0.2560717463493347]}, {"step": 2000, "epoch": 4, "train_loss": 0.07911437749862671, "cos_r_negg": [0.4253026247024536, 0.46877121925354004, 0.4506285786628723], "cos_apical_negg": [0.0011760637862607837, 0.04449416697025299, 0.10596850514411926], "cos_Ac_negg": [0.4329017698764801, 0.47436195611953735, 0.44663459062576294], "r_norm": [0.21458125114440918, 0.20688632130622864, 0.2070937305688858], "ldr": {"dL_sdil": -0.004229038953781128, "dL_bp": -0.004690013825893402, "ratio": 0.9017114044382454}}, {"step": 2100, "epoch": 4, "train_loss": 0.061470791697502136, "cos_r_negg": [0.4645769000053406, 0.5164486169815063, 0.49304959177970886], "cos_apical_negg": [0.022686440497636795, 0.061630189418792725, 0.12741413712501526], "cos_Ac_negg": [0.4717622399330139, 0.520348072052002, 0.486489862203598], "r_norm": [0.296880841255188, 0.28870129585266113, 0.28836122155189514]}, {"step": 2200, "epoch": 4, "train_loss": 0.008613009005784988, "cos_r_negg": [0.4541356563568115, 0.49870339035987854, 0.48270463943481445], "cos_apical_negg": [0.009809578768908978, 0.049802374094724655, 0.12591129541397095], "cos_Ac_negg": [0.46252015233039856, 0.504365086555481, 0.47725480794906616], "r_norm": [0.23834219574928284, 0.23024281859397888, 0.2296481430530548]}, {"step": 2300, "epoch": 4, "train_loss": 0.1314091831445694, "cos_r_negg": [0.434237539768219, 0.4750622510910034, 0.468572199344635], "cos_apical_negg": [0.012789031490683556, 0.056123536080121994, 0.12450282275676727], "cos_Ac_negg": [0.4436412453651428, 0.4804691672325134, 0.46218597888946533], "r_norm": [0.2297583520412445, 0.22174879908561707, 0.2221670299768448]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9701, "test_loss": 0.11755324440002442}, {"step": 2400, "epoch": 5, "train_loss": 0.0566553957760334, "cos_r_negg": [0.41152194142341614, 0.4643442928791046, 0.448253870010376], "cos_apical_negg": [0.0023106681182980537, 0.04468107223510742, 0.12120439112186432], "cos_Ac_negg": [0.4215308427810669, 0.4712718725204468, 0.4415430426597595], "r_norm": [0.19963353872299194, 0.1940317153930664, 0.19221720099449158]}, {"step": 2500, "epoch": 5, "train_loss": 0.02751944400370121, "cos_r_negg": [0.3762856423854828, 0.41709357500076294, 0.4034542441368103], "cos_apical_negg": [0.019218213856220245, 0.045613545924425125, 0.12064354866743088], "cos_Ac_negg": [0.3843798339366913, 0.42401909828186035, 0.3970809876918793], "r_norm": [0.21691936254501343, 0.21106751263141632, 0.2086942046880722], "ldr": {"dL_sdil": -0.0072431862354278564, "dL_bp": -0.009848549962043762, "ratio": 0.7354571244846239}}, {"step": 2600, "epoch": 5, "train_loss": 0.06003764271736145, "cos_r_negg": [0.3864571452140808, 0.4303845763206482, 0.42328310012817383], "cos_apical_negg": [0.024022221565246582, 0.045879196375608444, 0.1160430759191513], "cos_Ac_negg": [0.3941861689090729, 0.4359680116176605, 0.41608744859695435], "r_norm": [0.215977281332016, 0.21012946963310242, 0.20877155661582947]}, {"step": 2700, "epoch": 5, "train_loss": 0.03915666788816452, "cos_r_negg": [0.3820309042930603, 0.42565351724624634, 0.42161625623703003], "cos_apical_negg": [0.004506447818130255, 0.04175689071416855, 0.1131509467959404], "cos_Ac_negg": [0.3902479410171509, 0.4323440194129944, 0.4153056740760803], "r_norm": [0.18757474422454834, 0.18019750714302063, 0.18081000447273254]}, {"step": 2800, "epoch": 5, "train_loss": 0.11228148639202118, "cos_r_negg": [0.38285166025161743, 0.4258028268814087, 0.42557528614997864], "cos_apical_negg": [0.013368802145123482, 0.044960763305425644, 0.11763542890548706], "cos_Ac_negg": [0.39231181144714355, 0.4319199323654175, 0.41858869791030884], "r_norm": [0.23044544458389282, 0.22265984117984772, 0.22177085280418396]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9694, "test_loss": 0.1216586030960083}, {"step": 2900, "epoch": 6, "train_loss": 0.007296220399439335, "cos_r_negg": [0.3658027648925781, 0.4114435613155365, 0.40571752190589905], "cos_apical_negg": [0.009129548445343971, 0.044146232306957245, 0.11230024695396423], "cos_Ac_negg": [0.3764544725418091, 0.42097437381744385, 0.4017430245876312], "r_norm": [0.213870108127594, 0.20466497540473938, 0.20591892302036285]}, {"step": 3000, "epoch": 6, "train_loss": 0.08348645269870758, "cos_r_negg": [0.33541059494018555, 0.3902270793914795, 0.38683468103408813], "cos_apical_negg": [0.0019054042641073465, 0.040369823575019836, 0.11178232729434967], "cos_Ac_negg": [0.34923139214515686, 0.3980700969696045, 0.3826494514942169], "r_norm": [0.1901269108057022, 0.18310095369815826, 0.18356671929359436], "ldr": {"dL_sdil": -0.005525745451450348, "dL_bp": -0.00653473287820816, "ratio": 0.8455962247328341}}, {"step": 3100, "epoch": 6, "train_loss": 0.03265935927629471, "cos_r_negg": [0.3074042499065399, 0.35554981231689453, 0.35170480608940125], "cos_apical_negg": [0.013803176581859589, 0.041952282190322876, 0.11509326845407486], "cos_Ac_negg": [0.3213275671005249, 0.3644825220108032, 0.3470366597175598], "r_norm": [0.1561090350151062, 0.15100282430648804, 0.15105614066123962]}, {"step": 3200, "epoch": 6, "train_loss": 0.07671737670898438, "cos_r_negg": [0.3266666531562805, 0.3742595911026001, 0.3722819685935974], "cos_apical_negg": [0.008117439225316048, 0.040980592370033264, 0.11489686369895935], "cos_Ac_negg": [0.336247980594635, 0.3816106915473938, 0.36645907163619995], "r_norm": [0.19399002194404602, 0.18838225305080414, 0.18752212822437286]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9684, "test_loss": 0.13420426445007325}, {"step": 3300, "epoch": 7, "train_loss": 0.01861630193889141, "cos_r_negg": [0.32992905378341675, 0.37192773818969727, 0.36777275800704956], "cos_apical_negg": [0.006928750313818455, 0.03297814726829529, 0.10697699338197708], "cos_Ac_negg": [0.3389188051223755, 0.37956663966178894, 0.3609907031059265], "r_norm": [0.16771022975444794, 0.16327226161956787, 0.16160035133361816]}, {"step": 3400, "epoch": 7, "train_loss": 0.01749046891927719, "cos_r_negg": [0.28887417912483215, 0.33348017930984497, 0.3274837136268616], "cos_apical_negg": [0.0021591114345937967, 0.03380812332034111, 0.11148104071617126], "cos_Ac_negg": [0.2987444996833801, 0.34107738733291626, 0.32240432500839233], "r_norm": [0.189197838306427, 0.18410921096801758, 0.1830621063709259]}, {"step": 3500, "epoch": 7, "train_loss": 0.0033447002060711384, "cos_r_negg": [0.26638972759246826, 0.30873799324035645, 0.30631712079048157], "cos_apical_negg": [0.003221833147108555, 0.04037010297179222, 0.10763721913099289], "cos_Ac_negg": [0.278449684381485, 0.3174177408218384, 0.3028884530067444], "r_norm": [0.18936608731746674, 0.18346914649009705, 0.1843455582857132], "ldr": {"dL_sdil": -0.0070220232009887695, "dL_bp": -0.009894505143165588, "ratio": 0.7096891758997242}}, {"step": 3600, "epoch": 7, "train_loss": 0.007994148880243301, "cos_r_negg": [0.26776063442230225, 0.30211141705513, 0.2990608513355255], "cos_apical_negg": [0.0004425761289894581, 0.035594455897808075, 0.11064385622739792], "cos_Ac_negg": [0.27497997879981995, 0.30838608741760254, 0.29292333126068115], "r_norm": [0.17124144732952118, 0.16649122536182404, 0.16575351357460022]}, {"step": 3700, "epoch": 7, "train_loss": 0.01366973202675581, "cos_r_negg": [0.274452269077301, 0.31531739234924316, 0.3094955086708069], "cos_apical_negg": [0.0016667451709508896, 0.04143787547945976, 0.11769081652164459], "cos_Ac_negg": [0.28383803367614746, 0.3222227096557617, 0.30281198024749756], "r_norm": [0.19569653272628784, 0.18792316317558289, 0.18721522390842438]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9706, "test_loss": 0.12646207857131958}, {"step": 3800, "epoch": 8, "train_loss": 0.0028063447680324316, "cos_r_negg": [0.2572062909603119, 0.29839378595352173, 0.2979453206062317], "cos_apical_negg": [0.006120104342699051, 0.03888875991106033, 0.11478870362043381], "cos_Ac_negg": [0.2659004330635071, 0.306042343378067, 0.2925680875778198], "r_norm": [0.18202751874923706, 0.17508648335933685, 0.1745799034833908]}, {"step": 3900, "epoch": 8, "train_loss": 0.015280078165233135, "cos_r_negg": [0.23991478979587555, 0.2778204083442688, 0.27878713607788086], "cos_apical_negg": [0.004063733387738466, 0.0384351871907711, 0.1116466224193573], "cos_Ac_negg": [0.24855677783489227, 0.28675606846809387, 0.27356088161468506], "r_norm": [0.1674393117427826, 0.1606462597846985, 0.16081786155700684]}, {"step": 4000, "epoch": 8, "train_loss": 0.015317896381020546, "cos_r_negg": [0.24037721753120422, 0.2806762456893921, 0.27701348066329956], "cos_apical_negg": [-0.002131411572918296, 0.03127646446228027, 0.11113382875919342], "cos_Ac_negg": [0.25284796953201294, 0.29092222452163696, 0.273702472448349], "r_norm": [0.159669429063797, 0.15402299165725708, 0.1536632776260376], "ldr": {"dL_sdil": -0.005773581564426422, "dL_bp": -0.008616536855697632, "ratio": 0.6700582451067538}}, {"step": 4100, "epoch": 8, "train_loss": 0.008637665770947933, "cos_r_negg": [0.24601531028747559, 0.2782866060733795, 0.27356451749801636], "cos_apical_negg": [0.007809927687048912, 0.02853628247976303, 0.10848377645015717], "cos_Ac_negg": [0.25473126769065857, 0.28611791133880615, 0.26859909296035767], "r_norm": [0.1596459150314331, 0.15279893577098846, 0.15308859944343567]}, {"step": 4200, "epoch": 8, "train_loss": 0.013598673976957798, "cos_r_negg": [0.2147369682788849, 0.2427629679441452, 0.24022363126277924], "cos_apical_negg": [0.007440607994794846, 0.028577463701367378, 0.1035531535744667], "cos_Ac_negg": [0.22117993235588074, 0.25042492151260376, 0.23637595772743225], "r_norm": [0.153588205575943, 0.1472528874874115, 0.14731042087078094]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.9703, "test_loss": 0.1349515733718872}, {"step": 4300, "epoch": 9, "train_loss": 0.009483459405601025, "cos_r_negg": [0.19761964678764343, 0.22758793830871582, 0.2272750437259674], "cos_apical_negg": [0.01317617017775774, 0.030743807554244995, 0.10495631396770477], "cos_Ac_negg": [0.20658519864082336, 0.23675104975700378, 0.22355595231056213], "r_norm": [0.16664767265319824, 0.16051603853702545, 0.16062504053115845]}, {"step": 4400, "epoch": 9, "train_loss": 0.0009155111620202661, "cos_r_negg": [0.20603686571121216, 0.24109500646591187, 0.24256287515163422], "cos_apical_negg": [0.0055243270471692085, 0.03237554430961609, 0.10547778010368347], "cos_Ac_negg": [0.21358448266983032, 0.2480744570493698, 0.23845291137695312], "r_norm": [0.16977259516716003, 0.16366024315357208, 0.16286149621009827]}, {"step": 4500, "epoch": 9, "train_loss": 0.04439517855644226, "cos_r_negg": [0.19281518459320068, 0.22269965708255768, 0.2243371605873108], "cos_apical_negg": [0.001742643304169178, 0.025522764772176743, 0.09846746921539307], "cos_Ac_negg": [0.2023949921131134, 0.23147931694984436, 0.221358522772789], "r_norm": [0.14472535252571106, 0.13937348127365112, 0.1391879767179489], "ldr": {"dL_sdil": -0.005736790597438812, "dL_bp": -0.008923947811126709, "ratio": 0.6428534454544848}}, {"step": 4600, "epoch": 9, "train_loss": 0.00031514125294052064, "cos_r_negg": [0.1930336058139801, 0.22008375823497772, 0.22334088385105133], "cos_apical_negg": [-0.0008883214322850108, 0.025796813890337944, 0.10248971730470657], "cos_Ac_negg": [0.20047318935394287, 0.22753724455833435, 0.22068804502487183], "r_norm": [0.14989900588989258, 0.14418339729309082, 0.14354190230369568]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9723, "test_loss": 0.13656841354370117}, {"step": 4700, "epoch": 10, "train_loss": 0.003368678968399763, "cos_r_negg": [0.18275459110736847, 0.20915566384792328, 0.21093320846557617], "cos_apical_negg": [0.0038039106875658035, 0.02979060262441635, 0.09936227649450302], "cos_Ac_negg": [0.1908450722694397, 0.21591272950172424, 0.20760349929332733], "r_norm": [0.1625959277153015, 0.15653569996356964, 0.15662240982055664]}, {"step": 4800, "epoch": 10, "train_loss": 0.0036709927953779697, "cos_r_negg": [0.1759578287601471, 0.2027001678943634, 0.20458215475082397], "cos_apical_negg": [0.004462488926947117, 0.030056994408369064, 0.0965149849653244], "cos_Ac_negg": [0.18410442769527435, 0.2097683846950531, 0.20097805559635162], "r_norm": [0.15668311715126038, 0.15032222867012024, 0.15064573287963867]}, {"step": 4900, "epoch": 10, "train_loss": 0.0007979912916198373, "cos_r_negg": [0.17714136838912964, 0.2049698382616043, 0.20696143805980682], "cos_apical_negg": [0.005397228989750147, 0.030379384756088257, 0.09712393581867218], "cos_Ac_negg": [0.18587778508663177, 0.21150387823581696, 0.20400464534759521], "r_norm": [0.1544845700263977, 0.14891770482063293, 0.14883488416671753]}, {"step": 5000, "epoch": 10, "train_loss": 0.0011847037822008133, "cos_r_negg": [0.17362183332443237, 0.199512779712677, 0.20147936046123505], "cos_apical_negg": [0.002301005879417062, 0.028877578675746918, 0.09637881815433502], "cos_Ac_negg": [0.18046575784683228, 0.2065543830394745, 0.19924801588058472], "r_norm": [0.15203014016151428, 0.14576876163482666, 0.14624260365962982], "ldr": {"dL_sdil": -0.006801769137382507, "dL_bp": -0.011829875409603119, "ratio": 0.5749654076543398}}, {"step": 5100, "epoch": 10, "train_loss": 0.0039371708407998085, "cos_r_negg": [0.17761686444282532, 0.2053951621055603, 0.20478057861328125], "cos_apical_negg": [0.0023723230697214603, 0.029219835996627808, 0.09488993883132935], "cos_Ac_negg": [0.18559671938419342, 0.21271681785583496, 0.20284216105937958], "r_norm": [0.15508711338043213, 0.14951570332050323, 0.14984500408172607]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9724, "test_loss": 0.14021483154296874}, {"step": 5200, "epoch": 11, "train_loss": 0.0018229850102216005, "cos_r_negg": [0.16622629761695862, 0.19056853652000427, 0.19113439321517944], "cos_apical_negg": [0.002718140371143818, 0.026589151471853256, 0.0915985107421875], "cos_Ac_negg": [0.17470982670783997, 0.19878625869750977, 0.1902964562177658], "r_norm": [0.15578283369541168, 0.14982512593269348, 0.15023738145828247]}, {"step": 5300, "epoch": 11, "train_loss": 0.0029041029047220945, "cos_r_negg": [0.169172465801239, 0.19424086809158325, 0.19712267816066742], "cos_apical_negg": [0.006120515987277031, 0.026003392413258553, 0.09087199717760086], "cos_Ac_negg": [0.1776595413684845, 0.20163071155548096, 0.19550657272338867], "r_norm": [0.15783722698688507, 0.15186871588230133, 0.151752769947052]}, {"step": 5400, "epoch": 11, "train_loss": 0.0016632333863526583, "cos_r_negg": [0.16479048132896423, 0.18910513818264008, 0.19159580767154694], "cos_apical_negg": [0.004268251825124025, 0.02579345926642418, 0.09110432863235474], "cos_Ac_negg": [0.17256346344947815, 0.19645163416862488, 0.18975889682769775], "r_norm": [0.15863536298274994, 0.15256720781326294, 0.15291103720664978]}, {"step": 5500, "epoch": 11, "train_loss": 0.0014156047254800797, "cos_r_negg": [0.1590249389410019, 0.18435753881931305, 0.18659211695194244], "cos_apical_negg": [0.0027164453640580177, 0.026267051696777344, 0.09161141514778137], "cos_Ac_negg": [0.16864483058452606, 0.19191832840442657, 0.18541637063026428], "r_norm": [0.1609627604484558, 0.15514126420021057, 0.1552308350801468], "ldr": {"dL_sdil": -0.007866352796554565, "dL_bp": -0.015206754207611084, "ratio": 0.5172933480188299}}, {"step": 5600, "epoch": 11, "train_loss": 0.000706410501152277, "cos_r_negg": [0.15619783103466034, 0.17896431684494019, 0.18065303564071655], "cos_apical_negg": [-0.0008640203159302473, 0.027022305876016617, 0.09006054699420929], "cos_Ac_negg": [0.16408339142799377, 0.18575802445411682, 0.17898035049438477], "r_norm": [0.1536078155040741, 0.14769232273101807, 0.14831455051898956]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9723, "test_loss": 0.14282275924682616}, {"step": 5700, "epoch": 12, "train_loss": 0.0007750276708975434, "cos_r_negg": [0.151375412940979, 0.17433542013168335, 0.17635580897331238], "cos_apical_negg": [-0.0005568116903305054, 0.02616497501730919, 0.09106316417455673], "cos_Ac_negg": [0.15968874096870422, 0.18156278133392334, 0.17531467974185944], "r_norm": [0.16074919700622559, 0.15474316477775574, 0.15512396395206451]}, {"step": 5800, "epoch": 12, "train_loss": 0.00022004719357937574, "cos_r_negg": [0.1545858532190323, 0.17916078865528107, 0.18157872557640076], "cos_apical_negg": [-0.001574585447087884, 0.027285490185022354, 0.08827316761016846], "cos_Ac_negg": [0.16331452131271362, 0.1862182319164276, 0.18061158061027527], "r_norm": [0.15925638377666473, 0.15330514311790466, 0.15384294092655182]}, {"step": 5900, "epoch": 12, "train_loss": 0.00100623385515064, "cos_r_negg": [0.15426833927631378, 0.1789625585079193, 0.18174397945404053], "cos_apical_negg": [4.971446469426155e-05, 0.027911530807614326, 0.08503212034702301], "cos_Ac_negg": [0.16228662431240082, 0.18524077534675598, 0.18013641238212585], "r_norm": [0.15623131394386292, 0.1503482460975647, 0.15074537694454193]}, {"step": 6000, "epoch": 12, "train_loss": 0.0009307041764259338, "cos_r_negg": [0.1475083976984024, 0.17034894227981567, 0.17262369394302368], "cos_apical_negg": [0.0014557195827364922, 0.0270985197275877, 0.08524613082408905], "cos_Ac_negg": [0.15535986423492432, 0.17654819786548615, 0.1709788590669632], "r_norm": [0.15967300534248352, 0.15338923037052155, 0.15404221415519714], "ldr": {"dL_sdil": -0.008296497166156769, "dL_bp": -0.016536548733711243, "ratio": 0.5017066922340097}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9719, "test_loss": 0.14734946136474608}, {"step": 6100, "epoch": 13, "train_loss": 0.0007049450650811195, "cos_r_negg": [0.14751097559928894, 0.16981208324432373, 0.17184528708457947], "cos_apical_negg": [-0.0010662784334272146, 0.02532717026770115, 0.08324490487575531], "cos_Ac_negg": [0.15580761432647705, 0.17660905420780182, 0.17069850862026215], "r_norm": [0.15597400069236755, 0.15002527832984924, 0.15028820931911469]}, {"step": 6200, "epoch": 13, "train_loss": 0.0007785464404150844, "cos_r_negg": [0.14104576408863068, 0.16243742406368256, 0.16463854908943176], "cos_apical_negg": [-0.00039142509922385216, 0.02548239193856716, 0.08351120352745056], "cos_Ac_negg": [0.1496332287788391, 0.16892680525779724, 0.16343243420124054], "r_norm": [0.16047316789627075, 0.15447545051574707, 0.15490509569644928]}, {"step": 6300, "epoch": 13, "train_loss": 0.0009510027011856437, "cos_r_negg": [0.14160379767417908, 0.16322025656700134, 0.16558870673179626], "cos_apical_negg": [-2.2433116100728512e-05, 0.02655152603983879, 0.0823194831609726], "cos_Ac_negg": [0.14997750520706177, 0.16973009705543518, 0.16465634107589722], "r_norm": [0.16093528270721436, 0.15491116046905518, 0.15558366477489471]}, {"step": 6400, "epoch": 13, "train_loss": 0.000531387107912451, "cos_r_negg": [0.1414230763912201, 0.16334602236747742, 0.16590330004692078], "cos_apical_negg": [-0.00048621022142469883, 0.026712553575634956, 0.08361779153347015], "cos_Ac_negg": [0.14972209930419922, 0.16983509063720703, 0.1646653562784195], "r_norm": [0.16022735834121704, 0.15429165959358215, 0.1546449512243271]}, {"step": 6500, "epoch": 13, "train_loss": 0.00016070998390205204, "cos_r_negg": [0.14205138385295868, 0.16446661949157715, 0.1668013483285904], "cos_apical_negg": [-0.0032499597873538733, 0.02526477351784706, 0.08226311206817627], "cos_Ac_negg": [0.15034174919128418, 0.17092788219451904, 0.16568437218666077], "r_norm": [0.15671394765377045, 0.15060102939605713, 0.15110312402248383], "ldr": {"dL_sdil": -0.008396878838539124, "dL_bp": -0.017410598695278168, "ratio": 0.48228547366475083}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9718, "test_loss": 0.15054783363342286}, {"step": 6600, "epoch": 14, "train_loss": 0.00043517525773495436, "cos_r_negg": [0.13919903337955475, 0.1610000729560852, 0.16395319998264313], "cos_apical_negg": [-0.0008425507694482803, 0.025647016242146492, 0.08097261190414429], "cos_Ac_negg": [0.14758077263832092, 0.1673002541065216, 0.16278542578220367], "r_norm": [0.15801122784614563, 0.15237149596214294, 0.1524190902709961]}, {"step": 6700, "epoch": 14, "train_loss": 0.0002896473743021488, "cos_r_negg": [0.1386624127626419, 0.16038495302200317, 0.16338256001472473], "cos_apical_negg": [-0.0008073416538536549, 0.0265050008893013, 0.08070500940084457], "cos_Ac_negg": [0.14701414108276367, 0.1667577475309372, 0.1621738076210022], "r_norm": [0.15867236256599426, 0.15283624827861786, 0.1530551314353943]}, {"step": 6800, "epoch": 14, "train_loss": 0.0005891951732337475, "cos_r_negg": [0.1347474455833435, 0.15522809326648712, 0.15837156772613525], "cos_apical_negg": [-0.0024124933406710625, 0.026360051706433296, 0.08140113949775696], "cos_Ac_negg": [0.14297911524772644, 0.16109535098075867, 0.15688134729862213], "r_norm": [0.1590568721294403, 0.15323010087013245, 0.1533111333847046]}, {"step": 6900, "epoch": 14, "train_loss": 0.0006597358151338995, "cos_r_negg": [0.13322775065898895, 0.15402445197105408, 0.1572762280702591], "cos_apical_negg": [-0.0005752386059612036, 0.026457030326128006, 0.07867123186588287], "cos_Ac_negg": [0.14164724946022034, 0.15996012091636658, 0.1558118462562561], "r_norm": [0.15808779001235962, 0.1524420529603958, 0.15253888070583344]}, {"step": 7000, "epoch": 14, "train_loss": 0.0005647798534482718, "cos_r_negg": [0.13434326648712158, 0.15496627986431122, 0.1583033800125122], "cos_apical_negg": [-0.002938009798526764, 0.026398196816444397, 0.07914279401302338], "cos_Ac_negg": [0.14254990220069885, 0.16071441769599915, 0.15673944354057312], "r_norm": [0.16181103885173798, 0.15596053004264832, 0.15632116794586182], "ldr": {"dL_sdil": -0.00925840437412262, "dL_bp": -0.01928350329399109, "ratio": 0.4801204549283128}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9718, "test_loss": 0.15376302700042724}], "final": {"test_acc": 0.9718, "test_loss": 0.15376302700042724, "wall_s": 41.157554388046265, "cos_r_negg": [0.1340569257736206, 0.15481507778167725, 0.1580563336610794], "cos_apical_negg": [-0.003948848694562912, 0.025414934381842613, 0.07726860046386719], "cos_Ac_negg": [0.14251473546028137, 0.16108816862106323, 0.1570475995540619]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho1_res0_s0.json b/results/diag_nuis_v2_rho1_res0_s0.json new file mode 100644 index 0000000..aca3e63 --- /dev/null +++ b/results/diag_nuis_v2_rho1_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 1.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho1_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.0049112457782030106, -0.03490716964006424, 0.030286867171525955], "cos_apical_negg": [-0.003170457435771823, -0.030193276703357697, 0.004434592090547085], "cos_Ac_negg": [-0.004913521930575371, -0.03491111844778061, 0.030285123735666275], "r_norm": [4.915919303894043, 4.837163925170898, 4.831854820251465], "ldr": {"dL_sdil": 0.06604838371276855, "dL_bp": -0.17433524131774902, "ratio": -0.3788584752774492}}, {"step": 100, "epoch": 0, "train_loss": 6.168621063232422, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [5.298095703125, 5.2521562576293945, 5.305278301239014]}, {"step": 200, "epoch": 0, "train_loss": 7.308871269226074, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [5.07848596572876, 5.301395893096924, 5.309232711791992]}, {"step": 300, "epoch": 0, "train_loss": 9.080489158630371, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.773232460021973, 4.984652519226074, 4.902853965759277]}, {"step": 400, "epoch": 0, "train_loss": 8.882986068725586, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.050961971282959, 4.309518814086914, 4.272651672363281]}, {"epoch_end": 0, "step": 469, "test_acc": 0.1009, "test_loss": 7.86006201171875}, {"step": 500, "epoch": 1, "train_loss": 13.710022926330566, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.917295217514038, 4.270346641540527, 4.2291579246521], "ldr": {"dL_sdil": -1.8954887390136719, "dL_bp": -1.8954887390136719, "ratio": 1.0}}, {"step": 600, "epoch": 1, "train_loss": 12.950301170349121, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.137638568878174, 3.4815964698791504, 3.476419448852539]}, {"step": 700, "epoch": 1, "train_loss": 14.18130111694336, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.3657493591308594, 3.85614013671875, 3.901308536529541]}, {"step": 800, "epoch": 1, "train_loss": 8.794299125671387, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.1385321617126465, 3.464543342590332, 3.4572367668151855]}, {"step": 900, "epoch": 1, "train_loss": 9.17827033996582, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.433419704437256, 2.735952377319336, 2.8059771060943604]}, {"epoch_end": 1, "step": 938, "test_acc": 0.0982, "test_loss": 15.5167611328125}, {"step": 1000, "epoch": 2, "train_loss": 12.345236778259277, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.972174644470215, 3.3352460861206055, 3.4038267135620117], "ldr": {"dL_sdil": -2.452709197998047, "dL_bp": -2.452709197998047, "ratio": 1.0}}, {"step": 1100, "epoch": 2, "train_loss": 7.333985328674316, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.681196689605713, 2.9378514289855957, 2.991042137145996]}, {"step": 1200, "epoch": 2, "train_loss": 7.5777740478515625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.1562793254852295, 2.2864832878112793, 2.343010187149048]}, {"step": 1300, "epoch": 2, "train_loss": 12.29142951965332, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.377077102661133, 2.433556318283081, 2.6762166023254395]}, {"step": 1400, "epoch": 2, "train_loss": 14.754350662231445, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.080230712890625, 2.2352490425109863, 2.463550090789795]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.0892, "test_loss": 10.5706203125}, {"step": 1500, "epoch": 3, "train_loss": 7.104176044464111, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.8308072090148926, 1.902453899383545, 2.0528297424316406], "ldr": {"dL_sdil": -1.9092206954956055, "dL_bp": -1.9092206954956055, "ratio": 1.0}}, {"step": 1600, "epoch": 3, "train_loss": 14.409165382385254, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.678685188293457, 1.7203247547149658, 1.8382136821746826]}, {"step": 1700, "epoch": 3, "train_loss": 13.699381828308105, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6266570091247559, 1.7713520526885986, 1.922304630279541]}, {"step": 1800, "epoch": 3, "train_loss": 17.24734878540039, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5605814456939697, 1.6872401237487793, 1.8597053289413452]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.1009, "test_loss": 7.59026318359375}, {"step": 1900, "epoch": 4, "train_loss": 10.405152320861816, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7588977813720703, 1.9478669166564941, 2.0146589279174805]}, {"step": 2000, "epoch": 4, "train_loss": 7.993161678314209, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.252449870109558, 1.328159213066101, 1.4746103286743164], "ldr": {"dL_sdil": -1.083336353302002, "dL_bp": -1.083336353302002, "ratio": 1.0}}, {"step": 2100, "epoch": 4, "train_loss": 10.517053604125977, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5653631687164307, 1.6759041547775269, 1.9108890295028687]}, {"step": 2200, "epoch": 4, "train_loss": 16.935697555541992, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7173722982406616, 1.8259706497192383, 1.948004961013794]}, {"step": 2300, "epoch": 4, "train_loss": 10.041230201721191, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.3241275548934937, 1.3750216960906982, 1.5768585205078125]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.1009, "test_loss": 8.8302572265625}, {"step": 2400, "epoch": 5, "train_loss": 13.72265625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.1777900457382202, 1.2284111976623535, 1.4182350635528564]}, {"step": 2500, "epoch": 5, "train_loss": 12.191650390625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0629106760025024, 1.0615768432617188, 1.295640230178833], "ldr": {"dL_sdil": -2.5509347915649414, "dL_bp": -2.5509347915649414, "ratio": 1.0}}, {"step": 2600, "epoch": 5, "train_loss": 11.499530792236328, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.958137035369873, 2.0892257690429688, 2.21523380279541]}, {"step": 2700, "epoch": 5, "train_loss": 5.875406265258789, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9578269720077515, 0.9906907081604004, 1.2525248527526855]}, {"step": 2800, "epoch": 5, "train_loss": 8.57823657989502, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8988263607025146, 0.9357004165649414, 1.2335447072982788]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.1032, "test_loss": 10.96514287109375}, {"step": 2900, "epoch": 6, "train_loss": 11.10766315460205, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9529803395271301, 0.9400800466537476, 1.1965291500091553]}, {"step": 3000, "epoch": 6, "train_loss": 13.187132835388184, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8224517107009888, 0.8287823796272278, 1.1125569343566895], "ldr": {"dL_sdil": -1.3815803527832031, "dL_bp": -1.3815803527832031, "ratio": 1.0}}, {"step": 3100, "epoch": 6, "train_loss": 14.683507919311523, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7617363333702087, 0.786571741104126, 1.0990080833435059]}, {"step": 3200, "epoch": 6, "train_loss": 9.866296768188477, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.6981205344200134, 0.7051012516021729, 1.0233497619628906]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.098, "test_loss": 9.10851298828125}, {"step": 3300, "epoch": 7, "train_loss": 13.386348724365234, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6325187683105469, 1.6040260791778564, 1.888947606086731]}, {"step": 3400, "epoch": 7, "train_loss": 14.941996574401855, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.906234860420227, 0.9410721063613892, 1.2929428815841675]}, {"step": 3500, "epoch": 7, "train_loss": 14.875070571899414, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0326800346374512, 1.0611236095428467, 1.327538251876831], "ldr": {"dL_sdil": -1.154409408569336, "dL_bp": -1.154409408569336, "ratio": 1.0}}, {"step": 3600, "epoch": 7, "train_loss": 14.455398559570312, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9562624096870422, 0.9724542498588562, 1.374884009361267]}, {"step": 3700, "epoch": 7, "train_loss": 12.846620559692383, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7790042757987976, 0.7912087440490723, 1.1707276105880737]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.0982, "test_loss": 7.847995166015625}, {"step": 3800, "epoch": 8, "train_loss": 12.477538108825684, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.1448581218719482, 1.1362318992614746, 1.397266149520874]}, {"step": 3900, "epoch": 8, "train_loss": 12.69777774810791, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7793478965759277, 1.7625439167022705, 2.050631523132324]}, {"step": 4000, "epoch": 8, "train_loss": 6.235408782958984, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0840195417404175, 1.0865578651428223, 1.3329112529754639], "ldr": {"dL_sdil": -1.5645027160644531, "dL_bp": -1.5645027160644531, "ratio": 1.0}}, {"step": 4100, "epoch": 8, "train_loss": 8.821605682373047, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0421615839004517, 1.0486648082733154, 1.3295247554779053]}, {"step": 4200, "epoch": 8, "train_loss": 9.630763053894043, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.1512436866760254, 1.1475414037704468, 1.435883641242981]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.0974, "test_loss": 7.802023779296875}, {"step": 4300, "epoch": 9, "train_loss": 8.707650184631348, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7543022632598877, 1.7447994947433472, 2.016786813735962]}, {"step": 4400, "epoch": 9, "train_loss": 9.838920593261719, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0204123258590698, 1.0139331817626953, 1.3305269479751587]}, {"step": 4500, "epoch": 9, "train_loss": 8.983795166015625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.3118478059768677, 1.3124558925628662, 1.5741214752197266], "ldr": {"dL_sdil": -4.5032243728637695, "dL_bp": -4.5032243728637695, "ratio": 1.0}}, {"step": 4600, "epoch": 9, "train_loss": 13.195381164550781, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0432422161102295, 1.0458639860153198, 1.3557062149047852]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.0974, "test_loss": 10.37502734375}, {"step": 4700, "epoch": 10, "train_loss": 9.51148509979248, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.393178939819336, 1.3989429473876953, 1.6441538333892822]}, {"step": 4800, "epoch": 10, "train_loss": 10.038259506225586, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8864765167236328, 0.8864544630050659, 1.2801673412322998]}, {"step": 4900, "epoch": 10, "train_loss": 17.519451141357422, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7640345096588135, 1.7466367483139038, 1.9889378547668457]}, {"step": 5000, "epoch": 10, "train_loss": 9.583341598510742, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9011558294296265, 0.9014265537261963, 1.3205211162567139], "ldr": {"dL_sdil": -4.925800323486328, "dL_bp": -4.925800323486328, "ratio": 1.0}}, {"step": 5100, "epoch": 10, "train_loss": 12.114439010620117, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.6547254920005798, 0.6586419343948364, 1.0131325721740723]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.1135, "test_loss": 12.76181806640625}, {"step": 5200, "epoch": 11, "train_loss": 10.988119125366211, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.6414760947227478, 0.6437687873840332, 1.1603416204452515]}, {"step": 5300, "epoch": 11, "train_loss": 13.297741889953613, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6433727741241455, 1.6334799528121948, 1.8929803371429443]}, {"step": 5400, "epoch": 11, "train_loss": 10.380779266357422, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7482686638832092, 0.7499300241470337, 1.1333668231964111]}, {"step": 5500, "epoch": 11, "train_loss": 11.01572036743164, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.061265230178833, 1.0669736862182617, 1.357994794845581], "ldr": {"dL_sdil": -1.319023609161377, "dL_bp": -1.3190231323242188, "ratio": 1.0000003615078057}}, {"step": 5600, "epoch": 11, "train_loss": 8.639158248901367, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9644849896430969, 0.9690284132957458, 1.249232530593872]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.1028, "test_loss": 18.5220228515625}, {"step": 5700, "epoch": 12, "train_loss": 14.637080192565918, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.649249792098999, 0.6533929109573364, 1.1434708833694458]}, {"step": 5800, "epoch": 12, "train_loss": 5.07620906829834, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.5930944681167603, 0.5973938703536987, 1.0619299411773682]}, {"step": 5900, "epoch": 12, "train_loss": 5.498668193817139, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7330461144447327, 0.7312087416648865, 1.08412766456604]}, {"step": 6000, "epoch": 12, "train_loss": 19.36495590209961, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.185591459274292, 1.1945186853408813, 1.58848237991333], "ldr": {"dL_sdil": -5.1116533279418945, "dL_bp": -5.1116533279418945, "ratio": 1.0}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.1009, "test_loss": 9.9706404296875}, {"step": 6100, "epoch": 13, "train_loss": 11.17052936553955, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8670271039009094, 0.8683332204818726, 1.162795901298523]}, {"step": 6200, "epoch": 13, "train_loss": 9.35446548461914, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9485276937484741, 0.9506626725196838, 1.2338672876358032]}, {"step": 6300, "epoch": 13, "train_loss": 18.1002197265625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8742838501930237, 0.8760889172554016, 1.263365387916565]}, {"step": 6400, "epoch": 13, "train_loss": 12.906356811523438, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.964693546295166, 0.9625166654586792, 1.3287850618362427]}, {"step": 6500, "epoch": 13, "train_loss": 9.598943710327148, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.8981587290763855, 0.8989608287811279, 1.2203702926635742], "ldr": {"dL_sdil": -1.4609746932983398, "dL_bp": -1.4609746932983398, "ratio": 1.0}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.0982, "test_loss": 16.203428515625}, {"step": 6600, "epoch": 14, "train_loss": 14.59832763671875, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5515401363372803, 1.5561434030532837, 1.8323627710342407]}, {"step": 6700, "epoch": 14, "train_loss": 13.00601577758789, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7670224905014038, 0.7679570317268372, 1.1852943897247314]}, {"step": 6800, "epoch": 14, "train_loss": 9.41048526763916, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.7520909309387207, 0.751763105392456, 1.2077937126159668]}, {"step": 6900, "epoch": 14, "train_loss": 7.994902610778809, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.0834569931030273, 1.085732340812683, 1.3726246356964111]}, {"step": 7000, "epoch": 14, "train_loss": 12.404650688171387, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.6920681595802307, 0.6928187012672424, 1.1838388442993164], "ldr": {"dL_sdil": -6.562046051025391, "dL_bp": -6.562046051025391, "ratio": 1.0}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.0892, "test_loss": 11.24275126953125}], "final": {"test_acc": 0.0892, "test_loss": 11.24275126953125, "wall_s": 37.98674297332764, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho1_res1_s0.json b/results/diag_nuis_v2_rho1_res1_s0.json new file mode 100644 index 0000000..9e1258e --- /dev/null +++ b/results/diag_nuis_v2_rho1_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 1.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho1_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.01952069252729416, -0.025523580610752106, 0.03376835584640503], "cos_apical_negg": [-0.013020750135183334, -0.01925305277109146, 0.015020018443465233], "cos_Ac_negg": [-0.019519753754138947, -0.025523563846945763, 0.033768195658922195], "r_norm": [4.898232936859131, 4.8268141746521, 4.823862075805664], "ldr": {"dL_sdil": -0.009217500686645508, "dL_bp": -0.24074769020080566, "ratio": 0.03828697454566341}}, {"step": 100, "epoch": 0, "train_loss": 0.20820632576942444, "cos_r_negg": [0.45247799158096313, 0.3730752468109131, 0.3574973940849304], "cos_apical_negg": [0.05355706810951233, 0.05912838876247406, 0.11560722440481186], "cos_Ac_negg": [0.45980772376060486, 0.3796604573726654, 0.3623949885368347], "r_norm": [0.8269623517990112, 0.8108935356140137, 0.8083734512329102]}, {"step": 200, "epoch": 0, "train_loss": 0.3125769793987274, "cos_r_negg": [0.4705178439617157, 0.39170902967453003, 0.3797178864479065], "cos_apical_negg": [0.035528019070625305, 0.05168655514717102, 0.11112870275974274], "cos_Ac_negg": [0.48269566893577576, 0.4057140648365021, 0.394461989402771], "r_norm": [0.7476809620857239, 0.7304548621177673, 0.7294222116470337]}, {"step": 300, "epoch": 0, "train_loss": 0.17788541316986084, "cos_r_negg": [0.48671621084213257, 0.45401448011398315, 0.42767101526260376], "cos_apical_negg": [0.0389477014541626, 0.05002295598387718, 0.09516053646802902], "cos_Ac_negg": [0.4957050085067749, 0.4590369462966919, 0.43208760023117065], "r_norm": [0.5665696263313293, 0.5552384257316589, 0.5545287132263184]}, {"step": 400, "epoch": 0, "train_loss": 0.20246447622776031, "cos_r_negg": [0.5172643065452576, 0.46812668442726135, 0.4462651312351227], "cos_apical_negg": [0.03872915729880333, 0.05281463637948036, 0.10063380002975464], "cos_Ac_negg": [0.5260440111160278, 0.47443151473999023, 0.4496137499809265], "r_norm": [0.6011828780174255, 0.5863693952560425, 0.5862574577331543]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9489, "test_loss": 0.17657186698913574}, {"step": 500, "epoch": 1, "train_loss": 0.21730077266693115, "cos_r_negg": [0.4987039268016815, 0.48760223388671875, 0.4558370113372803], "cos_apical_negg": [0.026488296687602997, 0.05769956856966019, 0.10587222874164581], "cos_Ac_negg": [0.5154716968536377, 0.49703219532966614, 0.46710073947906494], "r_norm": [0.48388803005218506, 0.4662487506866455, 0.4681951105594635], "ldr": {"dL_sdil": -0.013195529580116272, "dL_bp": -0.012036949396133423, "ratio": 1.096251977627738}}, {"step": 600, "epoch": 1, "train_loss": 0.09636817872524261, "cos_r_negg": [0.49540555477142334, 0.4749462306499481, 0.4424026310443878], "cos_apical_negg": [0.024980027228593826, 0.045220229774713516, 0.1072184145450592], "cos_Ac_negg": [0.5133594274520874, 0.490916907787323, 0.456206738948822], "r_norm": [0.4222860336303711, 0.4137040376663208, 0.4124385714530945]}, {"step": 700, "epoch": 1, "train_loss": 0.20269888639450073, "cos_r_negg": [0.49148184061050415, 0.49356609582901, 0.46042340993881226], "cos_apical_negg": [0.023850711062550545, 0.05214633047580719, 0.10163506865501404], "cos_Ac_negg": [0.5164077281951904, 0.5084192752838135, 0.472796231508255], "r_norm": [0.5059418082237244, 0.4928421080112457, 0.49228620529174805]}, {"step": 800, "epoch": 1, "train_loss": 0.18580462038516998, "cos_r_negg": [0.5189096927642822, 0.5152329206466675, 0.476398229598999], "cos_apical_negg": [0.018776943907141685, 0.04151133447885513, 0.10171951353549957], "cos_Ac_negg": [0.5330885648727417, 0.5233570337295532, 0.4799470901489258], "r_norm": [0.35999131202697754, 0.34983736276626587, 0.34987181425094604]}, {"step": 900, "epoch": 1, "train_loss": 0.21851953864097595, "cos_r_negg": [0.5527595281600952, 0.5378736257553101, 0.47609639167785645], "cos_apical_negg": [0.020225191488862038, 0.03783454746007919, 0.09986966848373413], "cos_Ac_negg": [0.5599428415298462, 0.5394718050956726, 0.47620391845703125], "r_norm": [0.43943318724632263, 0.430154949426651, 0.42856162786483765]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9527, "test_loss": 0.15849240112304688}, {"step": 1000, "epoch": 2, "train_loss": 0.048953793942928314, "cos_r_negg": [0.5406612157821655, 0.5401836633682251, 0.4945063591003418], "cos_apical_negg": [0.00684305839240551, 0.03470243513584137, 0.1051340103149414], "cos_Ac_negg": [0.5539208054542542, 0.5516437888145447, 0.4969608783721924], "r_norm": [0.32489970326423645, 0.31662869453430176, 0.3150239884853363], "ldr": {"dL_sdil": -0.007094234228134155, "dL_bp": -0.006320945918560028, "ratio": 1.1223374348613775}}, {"step": 1100, "epoch": 2, "train_loss": 0.15747609734535217, "cos_r_negg": [0.5204079151153564, 0.5393697023391724, 0.4994724988937378], "cos_apical_negg": [0.01675550267100334, 0.03992713242769241, 0.10017327964305878], "cos_Ac_negg": [0.5327305793762207, 0.5504438877105713, 0.5013687014579773], "r_norm": [0.3689071536064148, 0.35580331087112427, 0.35768550634384155]}, {"step": 1200, "epoch": 2, "train_loss": 0.11255984008312225, "cos_r_negg": [0.5356619954109192, 0.5522195100784302, 0.5070484280586243], "cos_apical_negg": [0.020492661744356155, 0.02906835451722145, 0.0979154109954834], "cos_Ac_negg": [0.5454577207565308, 0.5596871376037598, 0.5072890520095825], "r_norm": [0.33136746287345886, 0.32186439633369446, 0.32262858748435974]}, {"step": 1300, "epoch": 2, "train_loss": 0.10070528090000153, "cos_r_negg": [0.5401700735092163, 0.5643941164016724, 0.49811404943466187], "cos_apical_negg": [0.014834541827440262, 0.04274377226829529, 0.11134599894285202], "cos_Ac_negg": [0.5524884462356567, 0.5701723694801331, 0.49451586604118347], "r_norm": [0.3253134489059448, 0.3205445110797882, 0.3168211281299591]}, {"step": 1400, "epoch": 2, "train_loss": 0.12846702337265015, "cos_r_negg": [0.5300706028938293, 0.5501449108123779, 0.4875084459781647], "cos_apical_negg": [0.022953219711780548, 0.03865577653050423, 0.10092078149318695], "cos_Ac_negg": [0.5395514369010925, 0.556358277797699, 0.48264631628990173], "r_norm": [0.3233911097049713, 0.3172415494918823, 0.3122522830963135]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9646, "test_loss": 0.11960855293273925}, {"step": 1500, "epoch": 3, "train_loss": 0.04802365228533745, "cos_r_negg": [0.5132083892822266, 0.5398122072219849, 0.48653846979141235], "cos_apical_negg": [0.004868463613092899, 0.04425199329853058, 0.09690235555171967], "cos_Ac_negg": [0.5291253328323364, 0.5514917969703674, 0.48411718010902405], "r_norm": [0.2663966417312622, 0.25964945554733276, 0.2593282163143158], "ldr": {"dL_sdil": -0.006809838116168976, "dL_bp": -0.007302872836589813, "ratio": 0.9324875659958681}}, {"step": 1600, "epoch": 3, "train_loss": 0.06981910765171051, "cos_r_negg": [0.5092813968658447, 0.5406162142753601, 0.5145689845085144], "cos_apical_negg": [0.007913642562925816, 0.04007811099290848, 0.10188837349414825], "cos_Ac_negg": [0.5224987864494324, 0.5543344020843506, 0.5140657424926758], "r_norm": [0.28141167759895325, 0.2739402651786804, 0.27224165201187134]}, {"step": 1700, "epoch": 3, "train_loss": 0.026035308837890625, "cos_r_negg": [0.50555419921875, 0.5345337390899658, 0.48899438977241516], "cos_apical_negg": [0.011308996006846428, 0.03269694000482559, 0.0982828140258789], "cos_Ac_negg": [0.5196442604064941, 0.5508270859718323, 0.49150747060775757], "r_norm": [0.2902778089046478, 0.2821534276008606, 0.28198111057281494]}, {"step": 1800, "epoch": 3, "train_loss": 0.04488299414515495, "cos_r_negg": [0.4855574071407318, 0.5212392807006836, 0.4940805435180664], "cos_apical_negg": [0.006815566681325436, 0.027186643332242966, 0.1029171422123909], "cos_Ac_negg": [0.5005114674568176, 0.5404747724533081, 0.4900423288345337], "r_norm": [0.24610629677772522, 0.23986393213272095, 0.23803485929965973]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9686, "test_loss": 0.11410523509979248}, {"step": 1900, "epoch": 4, "train_loss": 0.03720477595925331, "cos_r_negg": [0.48969316482543945, 0.5347305536270142, 0.5098276138305664], "cos_apical_negg": [0.006092734634876251, 0.03636404126882553, 0.10486426949501038], "cos_Ac_negg": [0.5025357604026794, 0.547871470451355, 0.5085148811340332], "r_norm": [0.2381521463394165, 0.2338624894618988, 0.23269914090633392]}, {"step": 2000, "epoch": 4, "train_loss": 0.09376846253871918, "cos_r_negg": [0.4365556240081787, 0.4885815978050232, 0.47137346863746643], "cos_apical_negg": [0.0021592299453914165, 0.03594357147812843, 0.10092498362064362], "cos_Ac_negg": [0.4539882242679596, 0.5067046284675598, 0.4683157801628113], "r_norm": [0.21684274077415466, 0.21048754453659058, 0.2079436033964157], "ldr": {"dL_sdil": -0.005363352596759796, "dL_bp": -0.00588611513376236, "ratio": 0.9111871709739361}}, {"step": 2100, "epoch": 4, "train_loss": 0.01795094832777977, "cos_r_negg": [0.4665408730506897, 0.5193837881088257, 0.49605539441108704], "cos_apical_negg": [0.010030766949057579, 0.04343145340681076, 0.10490712523460388], "cos_Ac_negg": [0.48169970512390137, 0.5332257747650146, 0.49485909938812256], "r_norm": [0.29260313510894775, 0.284512996673584, 0.28386205434799194]}, {"step": 2200, "epoch": 4, "train_loss": 0.008237517438828945, "cos_r_negg": [0.43938398361206055, 0.46865034103393555, 0.44824403524398804], "cos_apical_negg": [0.012633603997528553, 0.04007020592689514, 0.09947294741868973], "cos_Ac_negg": [0.4531918168067932, 0.4835525155067444, 0.44038569927215576], "r_norm": [0.2399400770664215, 0.2317260056734085, 0.23124423623085022]}, {"step": 2300, "epoch": 4, "train_loss": 0.11108306795358658, "cos_r_negg": [0.4207015633583069, 0.4533999562263489, 0.4320896863937378], "cos_apical_negg": [0.006102248094975948, 0.0322481133043766, 0.09669387340545654], "cos_Ac_negg": [0.433410108089447, 0.4679757356643677, 0.4283112585544586], "r_norm": [0.21574358642101288, 0.2103872150182724, 0.20982559025287628]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9693, "test_loss": 0.11552863693237304}, {"step": 2400, "epoch": 5, "train_loss": 0.08832602202892303, "cos_r_negg": [0.45157134532928467, 0.4912949204444885, 0.4594407379627228], "cos_apical_negg": [-0.0022061672061681747, 0.03439273312687874, 0.09876271337270737], "cos_Ac_negg": [0.46504831314086914, 0.5073860287666321, 0.45602723956108093], "r_norm": [0.20758934319019318, 0.202786386013031, 0.20093324780464172]}, {"step": 2500, "epoch": 5, "train_loss": 0.03226463124155998, "cos_r_negg": [0.3894649147987366, 0.429095983505249, 0.4108053147792816], "cos_apical_negg": [0.0025818003341555595, 0.031595081090927124, 0.10596092790365219], "cos_Ac_negg": [0.40533214807510376, 0.45035403966903687, 0.4087473452091217], "r_norm": [0.21533071994781494, 0.20972885191440582, 0.2089804708957672], "ldr": {"dL_sdil": -0.006639398634433746, "dL_bp": -0.008147932589054108, "ratio": 0.8148568439745171}}, {"step": 2600, "epoch": 5, "train_loss": 0.032373085618019104, "cos_r_negg": [0.40147143602371216, 0.4475758969783783, 0.43539121747016907], "cos_apical_negg": [-0.0051475162617862225, 0.023986084386706352, 0.1003330647945404], "cos_Ac_negg": [0.41641348600387573, 0.4666827619075775, 0.4289860725402832], "r_norm": [0.19734030961990356, 0.19379596412181854, 0.19054484367370605]}, {"step": 2700, "epoch": 5, "train_loss": 0.015110287815332413, "cos_r_negg": [0.4128378629684448, 0.4559454321861267, 0.4345250725746155], "cos_apical_negg": [0.01075565442442894, 0.034593287855386734, 0.09787613153457642], "cos_Ac_negg": [0.426885187625885, 0.477386474609375, 0.4317507743835449], "r_norm": [0.17885702848434448, 0.1742594689130783, 0.17370907962322235]}, {"step": 2800, "epoch": 5, "train_loss": 0.0966603234410286, "cos_r_negg": [0.3848547041416168, 0.4462057948112488, 0.42858922481536865], "cos_apical_negg": [-0.006684037856757641, 0.02884947508573532, 0.09844431281089783], "cos_Ac_negg": [0.40319156646728516, 0.46496981382369995, 0.4250919818878174], "r_norm": [0.2119465321302414, 0.20669062435626984, 0.2052062451839447]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.971, "test_loss": 0.11315425815582275}, {"step": 2900, "epoch": 6, "train_loss": 0.0055389562621712685, "cos_r_negg": [0.37317758798599243, 0.4125456213951111, 0.40699535608291626], "cos_apical_negg": [0.00035977899096906185, 0.02745770663022995, 0.10108094662427902], "cos_Ac_negg": [0.39138713479042053, 0.43325045704841614, 0.40558862686157227], "r_norm": [0.1856657862663269, 0.17919275164604187, 0.1803833544254303]}, {"step": 3000, "epoch": 6, "train_loss": 0.059122078120708466, "cos_r_negg": [0.35192257165908813, 0.3903104364871979, 0.38647234439849854], "cos_apical_negg": [0.00892309844493866, 0.03370555490255356, 0.10920710861682892], "cos_Ac_negg": [0.3734910488128662, 0.41287851333618164, 0.3839004635810852], "r_norm": [0.19960160553455353, 0.19387799501419067, 0.19266298413276672], "ldr": {"dL_sdil": -0.006923303008079529, "dL_bp": -0.009040042757987976, "ratio": 0.7658484803030328}}, {"step": 3100, "epoch": 6, "train_loss": 0.018755115568637848, "cos_r_negg": [0.3402131199836731, 0.3875967264175415, 0.3798831105232239], "cos_apical_negg": [0.00945977121591568, 0.036882299929857254, 0.10735878348350525], "cos_Ac_negg": [0.35401809215545654, 0.4052842855453491, 0.37230491638183594], "r_norm": [0.19227050244808197, 0.1867617517709732, 0.1859717220067978]}, {"step": 3200, "epoch": 6, "train_loss": 0.01891898550093174, "cos_r_negg": [0.31876227259635925, 0.35685282945632935, 0.36091458797454834], "cos_apical_negg": [0.0017555095255374908, 0.038343850523233414, 0.11688633263111115], "cos_Ac_negg": [0.3413740396499634, 0.38378798961639404, 0.35758328437805176], "r_norm": [0.20571985840797424, 0.2018389105796814, 0.19833488762378693]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.9692, "test_loss": 0.130955908203125}, {"step": 3300, "epoch": 7, "train_loss": 0.006738610565662384, "cos_r_negg": [0.31167787313461304, 0.3542804718017578, 0.3547123074531555], "cos_apical_negg": [0.010213890112936497, 0.03309930860996246, 0.10646143555641174], "cos_Ac_negg": [0.33059394359588623, 0.3761446475982666, 0.3523675799369812], "r_norm": [0.21223412454128265, 0.20840251445770264, 0.2042357176542282]}, {"step": 3400, "epoch": 7, "train_loss": 0.021810583770275116, "cos_r_negg": [0.27874547243118286, 0.3243314027786255, 0.3212192952632904], "cos_apical_negg": [-0.004605062305927277, 0.03871099278330803, 0.11275854706764221], "cos_Ac_negg": [0.3016117215156555, 0.3479066491127014, 0.32066595554351807], "r_norm": [0.23009563982486725, 0.2254328429698944, 0.2223561853170395]}, {"step": 3500, "epoch": 7, "train_loss": 0.008282961323857307, "cos_r_negg": [0.26876071095466614, 0.3095095753669739, 0.31059515476226807], "cos_apical_negg": [0.009344077669084072, 0.04012364149093628, 0.11219605803489685], "cos_Ac_negg": [0.28943419456481934, 0.330199658870697, 0.30517202615737915], "r_norm": [0.20065978169441223, 0.19628766179084778, 0.1945386677980423], "ldr": {"dL_sdil": -0.006579555571079254, "dL_bp": -0.009318768978118896, "ratio": 0.7060541565660119}}, {"step": 3600, "epoch": 7, "train_loss": 0.007229046430438757, "cos_r_negg": [0.27966195344924927, 0.3219839334487915, 0.320639967918396], "cos_apical_negg": [0.001273922622203827, 0.034568898379802704, 0.11067237704992294], "cos_Ac_negg": [0.30297043919563293, 0.3484964668750763, 0.32118597626686096], "r_norm": [0.20084795355796814, 0.1961166262626648, 0.19354857504367828]}, {"step": 3700, "epoch": 7, "train_loss": 0.0032357717864215374, "cos_r_negg": [0.2889690399169922, 0.3235850930213928, 0.32603615522384644], "cos_apical_negg": [0.005334996152669191, 0.030190210789442062, 0.1029483824968338], "cos_Ac_negg": [0.3125620186328888, 0.3513339161872864, 0.32547399401664734], "r_norm": [0.1978764533996582, 0.19294148683547974, 0.19131165742874146]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9718, "test_loss": 0.12202647438049316}, {"step": 3800, "epoch": 8, "train_loss": 0.004304760135710239, "cos_r_negg": [0.25752127170562744, 0.28930535912513733, 0.28976207971572876], "cos_apical_negg": [0.004450488369911909, 0.036352045834064484, 0.10605746507644653], "cos_Ac_negg": [0.27779683470726013, 0.3127424120903015, 0.28937214612960815], "r_norm": [0.19058868288993835, 0.1862567961215973, 0.18404096364974976]}, {"step": 3900, "epoch": 8, "train_loss": 0.021304629743099213, "cos_r_negg": [0.23813478648662567, 0.27522072196006775, 0.2766387462615967], "cos_apical_negg": [-0.00306108221411705, 0.041349273175001144, 0.10692794620990753], "cos_Ac_negg": [0.2591044008731842, 0.2974087595939636, 0.27524787187576294], "r_norm": [0.18729475140571594, 0.18307757377624512, 0.18050262331962585]}, {"step": 4000, "epoch": 8, "train_loss": 0.009143074974417686, "cos_r_negg": [0.22320961952209473, 0.26025712490081787, 0.26596689224243164], "cos_apical_negg": [-0.0003281468525528908, 0.03821016848087311, 0.108045294880867], "cos_Ac_negg": [0.24703139066696167, 0.28710439801216125, 0.26665085554122925], "r_norm": [0.17665278911590576, 0.17257598042488098, 0.17093247175216675], "ldr": {"dL_sdil": -0.007727883756160736, "dL_bp": -0.012262985110282898, "ratio": 0.6301796574539312}}, {"step": 4100, "epoch": 8, "train_loss": 0.015190917067229748, "cos_r_negg": [0.2306123524904251, 0.26366978883743286, 0.26940739154815674], "cos_apical_negg": [-0.0059013147838413715, 0.033381711691617966, 0.11045115441083908], "cos_Ac_negg": [0.24856306612491608, 0.2868881821632385, 0.26694488525390625], "r_norm": [0.17268167436122894, 0.1689237803220749, 0.16698026657104492]}, {"step": 4200, "epoch": 8, "train_loss": 0.02685653418302536, "cos_r_negg": [0.22369186580181122, 0.2567075490951538, 0.2605605721473694], "cos_apical_negg": [-0.0075320592150092125, 0.039410971105098724, 0.10670064389705658], "cos_Ac_negg": [0.2400529682636261, 0.2787850499153137, 0.25893548130989075], "r_norm": [0.18252675235271454, 0.17883211374282837, 0.17642846703529358]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.971, "test_loss": 0.13368471832275391}, {"step": 4300, "epoch": 9, "train_loss": 0.0005877374787814915, "cos_r_negg": [0.2049940526485443, 0.23769736289978027, 0.24234871566295624], "cos_apical_negg": [-0.016194412484765053, 0.040163516998291016, 0.09907465428113937], "cos_Ac_negg": [0.22388851642608643, 0.25845858454704285, 0.24186840653419495], "r_norm": [0.17549756169319153, 0.1730770766735077, 0.17001581192016602]}, {"step": 4400, "epoch": 9, "train_loss": 0.0025541128125041723, "cos_r_negg": [0.19466310739517212, 0.22876086831092834, 0.23361460864543915], "cos_apical_negg": [0.0013437815941870213, 0.039167940616607666, 0.10058873146772385], "cos_Ac_negg": [0.21199488639831543, 0.24881887435913086, 0.23262274265289307], "r_norm": [0.18421109020709991, 0.18011777102947235, 0.17842558026313782]}, {"step": 4500, "epoch": 9, "train_loss": 0.051683250814676285, "cos_r_negg": [0.18345549702644348, 0.2146534025669098, 0.21782127022743225], "cos_apical_negg": [0.0018967902287840843, 0.035548027604818344, 0.09891831129789352], "cos_Ac_negg": [0.20243990421295166, 0.23397545516490936, 0.21833869814872742], "r_norm": [0.19130754470825195, 0.18803565204143524, 0.1849699169397354], "ldr": {"dL_sdil": -0.00911315530538559, "dL_bp": -0.015487857162952423, "ratio": 0.5884064664016029}}, {"step": 4600, "epoch": 9, "train_loss": 0.0008189650252461433, "cos_r_negg": [0.20114709436893463, 0.23408544063568115, 0.2405523657798767], "cos_apical_negg": [-0.0015496102860197425, 0.032105863094329834, 0.10387919843196869], "cos_Ac_negg": [0.21795222163200378, 0.2518981993198395, 0.23951667547225952], "r_norm": [0.1851329207420349, 0.18125633895397186, 0.17906734347343445]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9718, "test_loss": 0.13972204494476317}, {"step": 4700, "epoch": 10, "train_loss": 0.001482734689489007, "cos_r_negg": [0.1834557205438614, 0.2140897661447525, 0.21974290907382965], "cos_apical_negg": [-0.0054072001948952675, 0.03423908352851868, 0.10417235642671585], "cos_Ac_negg": [0.19527043402194977, 0.2277383655309677, 0.21551403403282166], "r_norm": [0.18676964938640594, 0.1832066774368286, 0.18067359924316406]}, {"step": 4800, "epoch": 10, "train_loss": 0.023686068132519722, "cos_r_negg": [0.1680445671081543, 0.1973242163658142, 0.2018754482269287], "cos_apical_negg": [-0.008865216746926308, 0.03213939443230629, 0.09961983561515808], "cos_Ac_negg": [0.18008953332901, 0.21299338340759277, 0.20032998919487], "r_norm": [0.18621967732906342, 0.18277020752429962, 0.1804094910621643]}, {"step": 4900, "epoch": 10, "train_loss": 0.001245332881808281, "cos_r_negg": [0.1673194319009781, 0.19736751914024353, 0.20521703362464905], "cos_apical_negg": [-0.010764362290501595, 0.031070401892066002, 0.0963827520608902], "cos_Ac_negg": [0.18175038695335388, 0.21430855989456177, 0.2036517858505249], "r_norm": [0.18096566200256348, 0.17740479111671448, 0.17531053721904755]}, {"step": 5000, "epoch": 10, "train_loss": 0.0007794614648446441, "cos_r_negg": [0.16641497611999512, 0.1961652934551239, 0.20308934152126312], "cos_apical_negg": [-0.016297809779644012, 0.032492853701114655, 0.08998872339725494], "cos_Ac_negg": [0.18078342080116272, 0.21298450231552124, 0.20286217331886292], "r_norm": [0.18811345100402832, 0.1850011646747589, 0.18199634552001953], "ldr": {"dL_sdil": -0.00875924527645111, "dL_bp": -0.01467636227607727, "ratio": 0.596826728018893}}, {"step": 5100, "epoch": 10, "train_loss": 0.0015728764701634645, "cos_r_negg": [0.16495142877101898, 0.19899164140224457, 0.20289763808250427], "cos_apical_negg": [-0.008439348079264164, 0.0319247767329216, 0.09133325517177582], "cos_Ac_negg": [0.1828022301197052, 0.21493475139141083, 0.20316603779792786], "r_norm": [0.1848369538784027, 0.18157362937927246, 0.17898453772068024]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9719, "test_loss": 0.143025297164917}, {"step": 5200, "epoch": 11, "train_loss": 0.000945753650739789, "cos_r_negg": [0.15467843413352966, 0.1876443475484848, 0.19079965353012085], "cos_apical_negg": [-0.011053524911403656, 0.03400162607431412, 0.09057056903839111], "cos_Ac_negg": [0.17329344153404236, 0.2058325707912445, 0.19291263818740845], "r_norm": [0.18525457382202148, 0.18178372085094452, 0.17939798533916473]}, {"step": 5300, "epoch": 11, "train_loss": 0.0022470694966614246, "cos_r_negg": [0.15511254966259003, 0.18624407052993774, 0.19023746252059937], "cos_apical_negg": [-0.011065485887229443, 0.033132266253232956, 0.08913247287273407], "cos_Ac_negg": [0.16942650079727173, 0.2010449916124344, 0.1902359426021576], "r_norm": [0.18756461143493652, 0.18395178020000458, 0.18160291016101837]}, {"step": 5400, "epoch": 11, "train_loss": 0.0011650138767436147, "cos_r_negg": [0.15109407901763916, 0.18032802641391754, 0.18431201577186584], "cos_apical_negg": [-0.013389652594923973, 0.03147890791296959, 0.08431592583656311], "cos_Ac_negg": [0.16673095524311066, 0.19649699330329895, 0.18609780073165894], "r_norm": [0.18785886466503143, 0.18459686636924744, 0.1818501353263855]}, {"step": 5500, "epoch": 11, "train_loss": 0.0018065053736791015, "cos_r_negg": [0.15043359994888306, 0.18242962658405304, 0.18673600256443024], "cos_apical_negg": [-0.014000916853547096, 0.030169501900672913, 0.08574439585208893], "cos_Ac_negg": [0.16805027425289154, 0.1979711353778839, 0.18760564923286438], "r_norm": [0.18511494994163513, 0.18156656622886658, 0.17925119400024414], "ldr": {"dL_sdil": -0.009377487003803253, "dL_bp": -0.017397858202457428, "ratio": 0.5390023814815719}}, {"step": 5600, "epoch": 11, "train_loss": 0.0006028296193107963, "cos_r_negg": [0.14368405938148499, 0.1736426204442978, 0.17885416746139526], "cos_apical_negg": [-0.015546198934316635, 0.02911526709794998, 0.08045763522386551], "cos_Ac_negg": [0.15943577885627747, 0.1888650506734848, 0.17996357381343842], "r_norm": [0.17845308780670166, 0.17523901164531708, 0.17302536964416504]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9729, "test_loss": 0.1467744659423828}, {"step": 5700, "epoch": 12, "train_loss": 0.0018712173914536834, "cos_r_negg": [0.14311768114566803, 0.17202642560005188, 0.17876039445400238], "cos_apical_negg": [-0.013300701044499874, 0.028201447799801826, 0.07995752990245819], "cos_Ac_negg": [0.1575838029384613, 0.18676859140396118, 0.17903093993663788], "r_norm": [0.17752739787101746, 0.17435607314109802, 0.1720159649848938]}, {"step": 5800, "epoch": 12, "train_loss": 0.00026226392947137356, "cos_r_negg": [0.1410803496837616, 0.16962942481040955, 0.1758478432893753], "cos_apical_negg": [-0.016142388805747032, 0.029463719576597214, 0.08082856237888336], "cos_Ac_negg": [0.15645548701286316, 0.1850406527519226, 0.17691221833229065], "r_norm": [0.1738947331905365, 0.17098966240882874, 0.1685534417629242]}, {"step": 5900, "epoch": 12, "train_loss": 0.0019302917644381523, "cos_r_negg": [0.14157713949680328, 0.16993315517902374, 0.17557750642299652], "cos_apical_negg": [-0.01328299194574356, 0.029809674248099327, 0.07963322848081589], "cos_Ac_negg": [0.15610143542289734, 0.18432550132274628, 0.17637214064598083], "r_norm": [0.1819217950105667, 0.17911165952682495, 0.1762874722480774]}, {"step": 6000, "epoch": 12, "train_loss": 0.0006428231718018651, "cos_r_negg": [0.13808372616767883, 0.16538561880588531, 0.17137032747268677], "cos_apical_negg": [-0.014472590759396553, 0.02844945341348648, 0.07746273279190063], "cos_Ac_negg": [0.15271863341331482, 0.18086877465248108, 0.17349126935005188], "r_norm": [0.17917108535766602, 0.17625850439071655, 0.1736314296722412], "ldr": {"dL_sdil": -0.009854309260845184, "dL_bp": -0.018573276698589325, "ratio": 0.5305638536895129}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9727, "test_loss": 0.1511538637161255}, {"step": 6100, "epoch": 13, "train_loss": 0.0010399093152955174, "cos_r_negg": [0.13761809468269348, 0.16418057680130005, 0.16989046335220337], "cos_apical_negg": [-0.015085707418620586, 0.027884524315595627, 0.0746118426322937], "cos_Ac_negg": [0.15187600255012512, 0.17952817678451538, 0.17225655913352966], "r_norm": [0.18038342893123627, 0.17778252065181732, 0.17469583451747894]}, {"step": 6200, "epoch": 13, "train_loss": 0.0007959031499922276, "cos_r_negg": [0.1321205198764801, 0.15859249234199524, 0.16480687260627747], "cos_apical_negg": [-0.015651533380150795, 0.02816075086593628, 0.07136611640453339], "cos_Ac_negg": [0.14719578623771667, 0.17477412521839142, 0.16768872737884521], "r_norm": [0.17893503606319427, 0.1755964756011963, 0.17348116636276245]}, {"step": 6300, "epoch": 13, "train_loss": 0.0006927199428901076, "cos_r_negg": [0.13134489953517914, 0.15783515572547913, 0.16386264562606812], "cos_apical_negg": [-0.016916628926992416, 0.027650490403175354, 0.0720038115978241], "cos_Ac_negg": [0.14617937803268433, 0.1732787936925888, 0.16604876518249512], "r_norm": [0.18042808771133423, 0.17756028473377228, 0.1749606728553772]}, {"step": 6400, "epoch": 13, "train_loss": 0.0002556143736001104, "cos_r_negg": [0.13152113556861877, 0.15837058424949646, 0.16415289044380188], "cos_apical_negg": [-0.014501968398690224, 0.02755911275744438, 0.07234740257263184], "cos_Ac_negg": [0.14663293957710266, 0.1733982414007187, 0.16639921069145203], "r_norm": [0.18087442219257355, 0.17796151340007782, 0.17527058720588684]}, {"step": 6500, "epoch": 13, "train_loss": 3.9401020330842584e-05, "cos_r_negg": [0.12811042368412018, 0.15400443971157074, 0.15969786047935486], "cos_apical_negg": [-0.014461074024438858, 0.02677958272397518, 0.06982248276472092], "cos_Ac_negg": [0.14337000250816345, 0.16995283961296082, 0.1628904640674591], "r_norm": [0.17826606333255768, 0.1752840280532837, 0.17260578274726868], "ldr": {"dL_sdil": -0.01027720421552658, "dL_bp": -0.019783921539783478, "ratio": 0.5194725522369342}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9732, "test_loss": 0.15409321250915528}, {"step": 6600, "epoch": 14, "train_loss": 0.00021599166211672127, "cos_r_negg": [0.12744799256324768, 0.1532696783542633, 0.15923601388931274], "cos_apical_negg": [-0.014625275507569313, 0.026122236624360085, 0.06785876303911209], "cos_Ac_negg": [0.1434287577867508, 0.1701580137014389, 0.16322126984596252], "r_norm": [0.1740453839302063, 0.17141611874103546, 0.16868329048156738]}, {"step": 6700, "epoch": 14, "train_loss": 0.0005370760336518288, "cos_r_negg": [0.1292593777179718, 0.1553119719028473, 0.1612003743648529], "cos_apical_negg": [-0.014544112607836723, 0.02638331800699234, 0.06908605992794037], "cos_Ac_negg": [0.14379610121250153, 0.1703704297542572, 0.1642872393131256], "r_norm": [0.17507882416248322, 0.17225655913352966, 0.16960997879505157]}, {"step": 6800, "epoch": 14, "train_loss": 0.0007449829136021435, "cos_r_negg": [0.12631851434707642, 0.1522466540336609, 0.15829485654830933], "cos_apical_negg": [-0.014272957108914852, 0.02619188092648983, 0.06658457219600677], "cos_Ac_negg": [0.14164145290851593, 0.1677858680486679, 0.16174274682998657], "r_norm": [0.1792091578245163, 0.17650106549263, 0.17361101508140564]}, {"step": 6900, "epoch": 14, "train_loss": 0.00038480397779494524, "cos_r_negg": [0.12308157235383987, 0.14831876754760742, 0.1546904444694519], "cos_apical_negg": [-0.015797391533851624, 0.026111885905265808, 0.06552687287330627], "cos_Ac_negg": [0.13904376327991486, 0.16539886593818665, 0.15902471542358398], "r_norm": [0.1767159253358841, 0.17383286356925964, 0.17140065133571625]}, {"step": 7000, "epoch": 14, "train_loss": 0.0010645879665389657, "cos_r_negg": [0.12115307152271271, 0.14677995443344116, 0.15252578258514404], "cos_apical_negg": [-0.015390787273645401, 0.025602485984563828, 0.06431327015161514], "cos_Ac_negg": [0.13721109926700592, 0.16398105025291443, 0.15706533193588257], "r_norm": [0.17232541739940643, 0.16934022307395935, 0.16710829734802246], "ldr": {"dL_sdil": -0.009419582784175873, "dL_bp": -0.017572030425071716, "ratio": 0.5360554561034702}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9729, "test_loss": 0.15747193183898925}], "final": {"test_acc": 0.9729, "test_loss": 0.15747193183898925, "wall_s": 44.96389436721802, "cos_r_negg": [0.12099438905715942, 0.14610928297042847, 0.15199074149131775], "cos_apical_negg": [-0.015345013700425625, 0.025703568011522293, 0.0641845166683197], "cos_Ac_negg": [0.13689438998699188, 0.1636166274547577, 0.1564505398273468]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho2_res0_s0.json b/results/diag_nuis_v2_rho2_res0_s0.json new file mode 100644 index 0000000..4a23a63 --- /dev/null +++ b/results/diag_nuis_v2_rho2_res0_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 0, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 2.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho2_res0_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.011134970933198929, -0.04574815183877945, 0.019537901505827904], "cos_apical_negg": [-0.002000227104872465, -0.02902410179376602, -0.009386366233229637], "cos_Ac_negg": [-0.011133226566016674, -0.04575284570455551, 0.019536612555384636], "r_norm": [4.940460205078125, 4.8509297370910645, 4.84814977645874], "ldr": {"dL_sdil": -0.005567073822021484, "dL_bp": -0.18582582473754883, "ratio": 0.02995855839673599}}, {"step": 100, "epoch": 0, "train_loss": 5.65086555480957, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [5.942220211029053, 5.957629203796387, 5.864288330078125]}, {"step": 200, "epoch": 0, "train_loss": 11.260663032531738, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [5.360523223876953, 5.499752044677734, 5.407861709594727]}, {"step": 300, "epoch": 0, "train_loss": 9.852644920349121, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.545792579650879, 4.657017230987549, 4.558217525482178]}, {"step": 400, "epoch": 0, "train_loss": 7.6691575050354, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.124013900756836, 4.298738479614258, 4.182980537414551]}, {"epoch_end": 0, "step": 469, "test_acc": 0.0958, "test_loss": 11.84640537109375}, {"step": 500, "epoch": 1, "train_loss": 10.2719144821167, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.971134662628174, 4.165397644042969, 4.241818428039551], "ldr": {"dL_sdil": -4.683019638061523, "dL_bp": -4.683019638061523, "ratio": 1.0}}, {"step": 600, "epoch": 1, "train_loss": 11.482574462890625, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.664808511734009, 3.7119264602661133, 3.668844223022461]}, {"step": 700, "epoch": 1, "train_loss": 18.37065887451172, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.433459997177124, 3.532839298248291, 3.4711546897888184]}, {"step": 800, "epoch": 1, "train_loss": 17.872238159179688, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.411008596420288, 3.300015687942505, 3.2694523334503174]}, {"step": 900, "epoch": 1, "train_loss": 5.120199203491211, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.429213285446167, 3.5066628456115723, 3.702744483947754]}, {"epoch_end": 1, "step": 938, "test_acc": 0.101, "test_loss": 17.1151951171875}, {"step": 1000, "epoch": 2, "train_loss": 11.812946319580078, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.757107734680176, 2.799103260040283, 2.8523783683776855], "ldr": {"dL_sdil": -3.4034299850463867, "dL_bp": -3.4034290313720703, "ratio": 1.0000002802098436}}, {"step": 1100, "epoch": 2, "train_loss": 7.159360885620117, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.4967660903930664, 2.5222597122192383, 2.597177028656006]}, {"step": 1200, "epoch": 2, "train_loss": 14.094503402709961, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.4448676109313965, 2.4198811054229736, 2.465134620666504]}, {"step": 1300, "epoch": 2, "train_loss": 10.710535049438477, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.463955879211426, 2.4089598655700684, 2.5418879985809326]}, {"step": 1400, "epoch": 2, "train_loss": 9.26461410522461, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.0327658653259277, 1.9998868703842163, 2.1045291423797607]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.0892, "test_loss": 7.912837060546875}, {"step": 1500, "epoch": 3, "train_loss": 16.61450958251953, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.594482898712158, 2.4915390014648438, 2.686610221862793], "ldr": {"dL_sdil": -4.473783493041992, "dL_bp": -4.473783493041992, "ratio": 1.0}}, {"step": 1600, "epoch": 3, "train_loss": 10.878108978271484, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.383500576019287, 2.42948579788208, 2.5324554443359375]}, {"step": 1700, "epoch": 3, "train_loss": 13.330769538879395, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.9458767175674438, 1.9390082359313965, 2.053941249847412]}, {"step": 1800, "epoch": 3, "train_loss": 10.305810928344727, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.950385570526123, 1.9553329944610596, 2.0756518840789795]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.0892, "test_loss": 8.179347265625}, {"step": 1900, "epoch": 4, "train_loss": 9.913804054260254, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.966923475265503, 1.9365488290786743, 2.0935585498809814]}, {"step": 2000, "epoch": 4, "train_loss": 12.399497985839844, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7850117683410645, 1.8128559589385986, 1.9254733324050903], "ldr": {"dL_sdil": -2.467251777648926, "dL_bp": -2.467251777648926, "ratio": 1.0}}, {"step": 2100, "epoch": 4, "train_loss": 8.810676574707031, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7975695133209229, 1.7321829795837402, 1.846526861190796]}, {"step": 2200, "epoch": 4, "train_loss": 10.828862190246582, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7483348846435547, 1.6825271844863892, 1.8029508590698242]}, {"step": 2300, "epoch": 4, "train_loss": 14.336987495422363, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.4633861780166626, 1.4253790378570557, 1.557607650756836]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.0958, "test_loss": 8.21064892578125}, {"step": 2400, "epoch": 5, "train_loss": 17.109024047851562, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.3968760967254639, 1.3798121213912964, 1.5437554121017456]}, {"step": 2500, "epoch": 5, "train_loss": 18.47187614440918, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.1664862632751465, 2.077425956726074, 2.219440460205078], "ldr": {"dL_sdil": -2.3348731994628906, "dL_bp": -2.3348731994628906, "ratio": 1.0}}, {"step": 2600, "epoch": 5, "train_loss": 12.575115203857422, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.564220905303955, 2.641148567199707, 2.779033899307251]}, {"step": 2700, "epoch": 5, "train_loss": 17.852876663208008, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7907099723815918, 1.7467929124832153, 1.9062919616699219]}, {"step": 2800, "epoch": 5, "train_loss": 8.388150215148926, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.234530210494995, 2.2476446628570557, 2.37715220451355]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.0958, "test_loss": 9.38971689453125}, {"step": 2900, "epoch": 6, "train_loss": 5.755064964294434, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5859837532043457, 1.587763786315918, 1.830378532409668]}, {"step": 3000, "epoch": 6, "train_loss": 13.60433578491211, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.019087553024292, 2.0496044158935547, 2.2076194286346436], "ldr": {"dL_sdil": -3.2364463806152344, "dL_bp": -3.2364463806152344, "ratio": 1.0}}, {"step": 3100, "epoch": 6, "train_loss": 17.444042205810547, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.2913503646850586, 2.296858310699463, 2.437593460083008]}, {"step": 3200, "epoch": 6, "train_loss": 8.722667694091797, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.018972396850586, 3.0710816383361816, 3.23749041557312]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.1032, "test_loss": 9.35362158203125}, {"step": 3300, "epoch": 7, "train_loss": 19.265380859375, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.34082293510437, 2.3626480102539062, 2.5629613399505615]}, {"step": 3400, "epoch": 7, "train_loss": 9.322088241577148, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.101423501968384, 2.118718385696411, 2.3106136322021484]}, {"step": 3500, "epoch": 7, "train_loss": 10.016507148742676, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.61253023147583, 2.6426291465759277, 2.8267617225646973], "ldr": {"dL_sdil": -3.567035675048828, "dL_bp": -3.567035675048828, "ratio": 1.0}}, {"step": 3600, "epoch": 7, "train_loss": 8.033374786376953, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.984973430633545, 3.0006351470947266, 3.2030181884765625]}, {"step": 3700, "epoch": 7, "train_loss": 9.016714096069336, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.289839267730713, 2.2974023818969727, 2.5099151134490967]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.0982, "test_loss": 10.193218359375}, {"step": 3800, "epoch": 8, "train_loss": 10.779725074768066, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.169066905975342, 4.192922592163086, 4.302099227905273]}, {"step": 3900, "epoch": 8, "train_loss": 13.918378829956055, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.965263843536377, 1.9798157215118408, 2.151416301727295]}, {"step": 4000, "epoch": 8, "train_loss": 10.645133018493652, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.0207700729370117, 2.032172203063965, 2.1995136737823486], "ldr": {"dL_sdil": -3.1146554946899414, "dL_bp": -3.1146554946899414, "ratio": 1.0}}, {"step": 4100, "epoch": 8, "train_loss": 13.109430313110352, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.1667044162750244, 2.1754820346832275, 2.4026706218719482]}, {"step": 4200, "epoch": 8, "train_loss": 15.165124893188477, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.9740670919418335, 1.99790620803833, 2.1787705421447754]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.1135, "test_loss": 11.08155712890625}, {"step": 4300, "epoch": 9, "train_loss": 6.7026777267456055, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.56621515750885, 1.5709431171417236, 1.7454463243484497]}, {"step": 4400, "epoch": 9, "train_loss": 12.094133377075195, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.8710389137268066, 1.8770842552185059, 2.080204486846924]}, {"step": 4500, "epoch": 9, "train_loss": 9.14280891418457, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.8006874322891235, 1.8123821020126343, 1.9861721992492676], "ldr": {"dL_sdil": -4.5993547439575195, "dL_bp": -4.5993547439575195, "ratio": 1.0}}, {"step": 4600, "epoch": 9, "train_loss": 10.847969055175781, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.2693443298339844, 2.2822179794311523, 2.43780255317688]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.0892, "test_loss": 13.40768349609375}, {"step": 4700, "epoch": 10, "train_loss": 13.125690460205078, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7040889263153076, 1.7094494104385376, 1.9003942012786865]}, {"step": 4800, "epoch": 10, "train_loss": 15.025775909423828, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6433634757995605, 1.6440839767456055, 1.9935144186019897]}, {"step": 4900, "epoch": 10, "train_loss": 14.747944831848145, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7956981658935547, 1.7881911993026733, 2.17130184173584]}, {"step": 5000, "epoch": 10, "train_loss": 7.381263732910156, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.341409683227539, 2.342508316040039, 2.5071516036987305], "ldr": {"dL_sdil": -1.8600077629089355, "dL_bp": -1.8600077629089355, "ratio": 1.0}}, {"step": 5100, "epoch": 10, "train_loss": 7.922386169433594, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [0.9084875583648682, 0.9084047675132751, 1.3027852773666382]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.0974, "test_loss": 7.684931103515625}, {"step": 5200, "epoch": 11, "train_loss": 9.788581848144531, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.8124029636383057, 1.8253060579299927, 2.0523672103881836]}, {"step": 5300, "epoch": 11, "train_loss": 6.008083343505859, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.537026286125183, 1.5404973030090332, 1.7582496404647827]}, {"step": 5400, "epoch": 11, "train_loss": 10.358059883117676, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.3209798336029053, 1.3235864639282227, 1.601078987121582]}, {"step": 5500, "epoch": 11, "train_loss": 8.494882583618164, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.080670952796936, 1.085646629333496, 1.4499411582946777], "ldr": {"dL_sdil": -2.2096853256225586, "dL_bp": -2.2096853256225586, "ratio": 1.0}}, {"step": 5600, "epoch": 11, "train_loss": 14.80677604675293, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6960722208023071, 1.695042610168457, 1.9365251064300537]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.1135, "test_loss": 8.34606416015625}, {"step": 5700, "epoch": 12, "train_loss": 9.628883361816406, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.5165657997131348, 3.532578945159912, 3.645426034927368]}, {"step": 5800, "epoch": 12, "train_loss": 15.433618545532227, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5733294486999512, 1.5801420211791992, 1.7934064865112305]}, {"step": 5900, "epoch": 12, "train_loss": 11.666193008422852, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [3.6554133892059326, 3.6720900535583496, 3.787010431289673]}, {"step": 6000, "epoch": 12, "train_loss": 13.397811889648438, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7953414916992188, 1.802546739578247, 1.993793249130249], "ldr": {"dL_sdil": -2.438274383544922, "dL_bp": -2.438274383544922, "ratio": 1.0}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.0958, "test_loss": 8.74623095703125}, {"step": 6100, "epoch": 13, "train_loss": 12.094367980957031, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.4503973722457886, 1.4523446559906006, 1.7613974809646606]}, {"step": 6200, "epoch": 13, "train_loss": 15.628083229064941, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.5844041109085083, 1.5868403911590576, 1.8414292335510254]}, {"step": 6300, "epoch": 13, "train_loss": 8.644603729248047, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.3171517848968506, 1.3178538084030151, 1.5690869092941284]}, {"step": 6400, "epoch": 13, "train_loss": 11.285953521728516, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.09321665763855, 2.0986382961273193, 2.3055248260498047]}, {"step": 6500, "epoch": 13, "train_loss": 11.769700050354004, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.7721292972564697, 1.775016188621521, 1.9729554653167725], "ldr": {"dL_sdil": -1.5939130783081055, "dL_bp": -1.5939130783081055, "ratio": 1.0}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.0958, "test_loss": 13.54838427734375}, {"step": 6600, "epoch": 14, "train_loss": 15.937044143676758, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.4370322227478027, 2.438749074935913, 2.6168036460876465]}, {"step": 6700, "epoch": 14, "train_loss": 8.503223419189453, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [2.8197872638702393, 2.8202476501464844, 2.979583501815796]}, {"step": 6800, "epoch": 14, "train_loss": 10.037688255310059, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.6860488653182983, 1.6880760192871094, 1.9711626768112183]}, {"step": 6900, "epoch": 14, "train_loss": 7.726430416107178, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [1.697356939315796, 1.6979938745498657, 1.97635817527771]}, {"step": 7000, "epoch": 14, "train_loss": 15.554264068603516, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0], "r_norm": [4.120566368103027, 4.127189636230469, 4.276511192321777], "ldr": {"dL_sdil": -1.9065895080566406, "dL_bp": -1.9065895080566406, "ratio": 1.0}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.0982, "test_loss": 9.38495908203125}], "final": {"test_acc": 0.0982, "test_loss": 9.38495908203125, "wall_s": 41.892329454422, "cos_r_negg": [0.0, 0.0, 0.0], "cos_apical_negg": [0.0, 0.0, 0.0], "cos_Ac_negg": [0.0, 0.0, 0.0]}}
\ No newline at end of file diff --git a/results/diag_nuis_v2_rho2_res1_s0.json b/results/diag_nuis_v2_rho2_res1_s0.json new file mode 100644 index 0000000..43a872d --- /dev/null +++ b/results/diag_nuis_v2_rho2_res1_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "mnist", "depth": 3, "width": 256, "act": "tanh", "residual": 0, "epochs": 15, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 2.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 100, "max_steps": 0, "probe_bs": 512, "outdir": "results", "tag": "diag_nuis_v2_rho2_res1_s0", "n_in": 784}, "provenance": {"git_commit": "54633b2e0f51a8f48d4ba627d778c87927f54ca1", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.3603768348693848, "cos_r_negg": [-0.019525395706295967, -0.02552098035812378, 0.03377208486199379], "cos_apical_negg": [-0.008897066116333008, -0.013789067044854164, 0.004767646547406912], "cos_Ac_negg": [-0.019523516297340393, -0.02552095055580139, 0.033771760761737823], "r_norm": [4.8982954025268555, 4.826826095581055, 4.823868751525879], "ldr": {"dL_sdil": -0.009236812591552734, "dL_bp": -0.24081993103027344, "ratio": 0.03835568157517484}}, {"step": 100, "epoch": 0, "train_loss": 0.2104795277118683, "cos_r_negg": [0.4429137110710144, 0.3625786602497101, 0.3501937985420227], "cos_apical_negg": [0.02913225069642067, 0.0413375049829483, 0.09548983722925186], "cos_Ac_negg": [0.4604405462741852, 0.3800952434539795, 0.36436647176742554], "r_norm": [0.8362362384796143, 0.8200146555900574, 0.817680299282074]}, {"step": 200, "epoch": 0, "train_loss": 0.2786995768547058, "cos_r_negg": [0.46058589220046997, 0.38203614950180054, 0.3677726984024048], "cos_apical_negg": [0.014530155807733536, 0.041846051812171936, 0.09950847923755646], "cos_Ac_negg": [0.49076348543167114, 0.4162990152835846, 0.3995876908302307], "r_norm": [0.7659614086151123, 0.7472445964813232, 0.7463458776473999]}, {"step": 300, "epoch": 0, "train_loss": 0.1681954711675644, "cos_r_negg": [0.5001245737075806, 0.4624118208885193, 0.4346352815628052], "cos_apical_negg": [0.02112080529332161, 0.03908928483724594, 0.08193816989660263], "cos_Ac_negg": [0.5147418975830078, 0.4728078842163086, 0.43891194462776184], "r_norm": [0.5594145059585571, 0.5477821826934814, 0.5476260781288147]}, {"step": 400, "epoch": 0, "train_loss": 0.1623169183731079, "cos_r_negg": [0.5148628950119019, 0.48024970293045044, 0.430259644985199], "cos_apical_negg": [0.019099585711956024, 0.029904361814260483, 0.07807426899671555], "cos_Ac_negg": [0.5364471077919006, 0.4963504672050476, 0.4369904696941376], "r_norm": [0.5310893654823303, 0.5209400653839111, 0.5205697417259216]}, {"epoch_end": 0, "step": 469, "test_acc": 0.9452, "test_loss": 0.18473090362548827}, {"step": 500, "epoch": 1, "train_loss": 0.2528912127017975, "cos_r_negg": [0.4793739914894104, 0.46812039613723755, 0.4326707124710083], "cos_apical_negg": [0.014721769839525223, 0.04298064857721329, 0.09922119975090027], "cos_Ac_negg": [0.5140164494514465, 0.4957870841026306, 0.45290637016296387], "r_norm": [0.534934401512146, 0.5158824920654297, 0.5166623592376709], "ldr": {"dL_sdil": -0.018173053860664368, "dL_bp": -0.016316741704940796, "ratio": 1.1137673310818832}}, {"step": 600, "epoch": 1, "train_loss": 0.09457680583000183, "cos_r_negg": [0.48122870922088623, 0.4665687680244446, 0.43470096588134766], "cos_apical_negg": [0.00775455217808485, 0.030592598021030426, 0.09659455716609955], "cos_Ac_negg": [0.5211817026138306, 0.49690839648246765, 0.46077242493629456], "r_norm": [0.4056410491466522, 0.39857375621795654, 0.39543288946151733]}, {"step": 700, "epoch": 1, "train_loss": 0.18847103416919708, "cos_r_negg": [0.465390682220459, 0.4821118116378784, 0.45879414677619934], "cos_apical_negg": [0.003821765072643757, 0.04252293333411217, 0.1065407395362854], "cos_Ac_negg": [0.515001654624939, 0.5152771472930908, 0.4849601984024048], "r_norm": [0.4375430941581726, 0.42695391178131104, 0.4247203469276428]}, {"step": 800, "epoch": 1, "train_loss": 0.16579097509384155, "cos_r_negg": [0.508460283279419, 0.5228742361068726, 0.47062426805496216], "cos_apical_negg": [0.01265665516257286, 0.04129909351468086, 0.10414852201938629], "cos_Ac_negg": [0.541027843952179, 0.5396636724472046, 0.48390957713127136], "r_norm": [0.3352852463722229, 0.3273393213748932, 0.32507359981536865]}, {"step": 900, "epoch": 1, "train_loss": 0.18073377013206482, "cos_r_negg": [0.5507323741912842, 0.5638676881790161, 0.48661351203918457], "cos_apical_negg": [0.003939080983400345, 0.0321526899933815, 0.09565563499927521], "cos_Ac_negg": [0.5651798248291016, 0.5708035230636597, 0.48854005336761475], "r_norm": [0.3477749824523926, 0.3410939872264862, 0.33893710374832153]}, {"epoch_end": 1, "step": 938, "test_acc": 0.9545, "test_loss": 0.14937169456481933}, {"step": 1000, "epoch": 2, "train_loss": 0.05696079134941101, "cos_r_negg": [0.503943681716919, 0.5188122987747192, 0.47791409492492676], "cos_apical_negg": [0.005241388455033302, 0.03323132544755936, 0.09120991826057434], "cos_Ac_negg": [0.5342360138893127, 0.5386021137237549, 0.48692798614501953], "r_norm": [0.30671465396881104, 0.29946863651275635, 0.29653069376945496], "ldr": {"dL_sdil": -0.008440732955932617, "dL_bp": -0.007455542683601379, "ratio": 1.132141993432374}}, {"step": 1100, "epoch": 2, "train_loss": 0.09730264544487, "cos_r_negg": [0.4988236427307129, 0.5263614058494568, 0.47083091735839844], "cos_apical_negg": [0.01208480540663004, 0.04290391132235527, 0.08295844495296478], "cos_Ac_negg": [0.5341017246246338, 0.5504165887832642, 0.4866880178451538], "r_norm": [0.35640671849250793, 0.3450624346733093, 0.3451017439365387]}, {"step": 1200, "epoch": 2, "train_loss": 0.1298961490392685, "cos_r_negg": [0.5133548974990845, 0.5411757826805115, 0.4817139804363251], "cos_apical_negg": [0.0056552765890955925, 0.03371363878250122, 0.08027148246765137], "cos_Ac_negg": [0.5373932123184204, 0.5554259419441223, 0.4888988137245178], "r_norm": [0.3141748905181885, 0.3061453700065613, 0.3053479790687561]}, {"step": 1300, "epoch": 2, "train_loss": 0.14578621089458466, "cos_r_negg": [0.5057678818702698, 0.528052806854248, 0.4672167897224426], "cos_apical_negg": [0.003955305553972721, 0.039266444742679596, 0.09113456308841705], "cos_Ac_negg": [0.535873532295227, 0.5495326519012451, 0.4734228253364563], "r_norm": [0.32081305980682373, 0.3140150308609009, 0.311356782913208]}, {"step": 1400, "epoch": 2, "train_loss": 0.15383687615394592, "cos_r_negg": [0.47028887271881104, 0.5024353861808777, 0.45877429842948914], "cos_apical_negg": [-0.000482750590890646, 0.043554145842790604, 0.09052295982837677], "cos_Ac_negg": [0.4933069348335266, 0.5139294266700745, 0.46155816316604614], "r_norm": [0.28868618607521057, 0.28035980463027954, 0.2792937159538269]}, {"epoch_end": 2, "step": 1407, "test_acc": 0.9641, "test_loss": 0.12704081268310546}, {"step": 1500, "epoch": 3, "train_loss": 0.07152651250362396, "cos_r_negg": [0.46427416801452637, 0.5024504661560059, 0.4708280861377716], "cos_apical_negg": [0.005512584000825882, 0.04187152534723282, 0.09414386749267578], "cos_Ac_negg": [0.5033773183822632, 0.5319558382034302, 0.4813491702079773], "r_norm": [0.23926027119159698, 0.23365998268127441, 0.2324751913547516], "ldr": {"dL_sdil": -0.004995793104171753, "dL_bp": -0.004729658365249634, "ratio": 1.0562693366771476}}, {"step": 1600, "epoch": 3, "train_loss": 0.0974859744310379, "cos_r_negg": [0.4652981460094452, 0.5063573122024536, 0.48069697618484497], "cos_apical_negg": [-0.008264572359621525, 0.03116120584309101, 0.09904827177524567], "cos_Ac_negg": [0.5057864189147949, 0.541628360748291, 0.49891218543052673], "r_norm": [0.2457239031791687, 0.23975686728954315, 0.23840028047561646]}, {"step": 1700, "epoch": 3, "train_loss": 0.0348355695605278, "cos_r_negg": [0.47419023513793945, 0.513866662979126, 0.48939383029937744], "cos_apical_negg": [-0.009268120862543583, 0.03703146427869797, 0.10172654688358307], "cos_Ac_negg": [0.5040693879127502, 0.5393393039703369, 0.49916988611221313], "r_norm": [0.2812294661998749, 0.27222853899002075, 0.2717058062553406]}, {"step": 1800, "epoch": 3, "train_loss": 0.021023377776145935, "cos_r_negg": [0.465406596660614, 0.5011543035507202, 0.4807687997817993], "cos_apical_negg": [-0.002072057453915477, 0.028440648689866066, 0.08577912300825119], "cos_Ac_negg": [0.5046777725219727, 0.5338564515113831, 0.49394911527633667], "r_norm": [0.21957169473171234, 0.21516944468021393, 0.2121545374393463]}, {"epoch_end": 3, "step": 1876, "test_acc": 0.9674, "test_loss": 0.1202873867034912}, {"step": 1900, "epoch": 4, "train_loss": 0.02093462646007538, "cos_r_negg": [0.4647122025489807, 0.5101388096809387, 0.4865616261959076], "cos_apical_negg": [-0.005798174999654293, 0.03767603635787964, 0.08937495946884155], "cos_Ac_negg": [0.5050925612449646, 0.5463800430297852, 0.5008324384689331], "r_norm": [0.2384132444858551, 0.2347446233034134, 0.23052723705768585]}, {"step": 2000, "epoch": 4, "train_loss": 0.10467495024204254, "cos_r_negg": [0.3863235116004944, 0.4237010180950165, 0.4237331748008728], "cos_apical_negg": [-0.01224249042570591, 0.03634224086999893, 0.09541654586791992], "cos_Ac_negg": [0.4352112412452698, 0.4720419943332672, 0.44034767150878906], "r_norm": [0.16932746767997742, 0.16426919400691986, 0.16319605708122253], "ldr": {"dL_sdil": -0.003473937511444092, "dL_bp": -0.0037461817264556885, "ratio": 0.9273275471157747}}, {"step": 2100, "epoch": 4, "train_loss": 0.017748109996318817, "cos_r_negg": [0.4375082850456238, 0.48254895210266113, 0.47639352083206177], "cos_apical_negg": [0.010370379313826561, 0.03625763952732086, 0.09920184314250946], "cos_Ac_negg": [0.4788301885128021, 0.5247420072555542, 0.494262158870697], "r_norm": [0.20781828463077545, 0.20187830924987793, 0.20186066627502441]}, {"step": 2200, "epoch": 4, "train_loss": 0.012651554308831692, "cos_r_negg": [0.40401971340179443, 0.4583597183227539, 0.4443907141685486], "cos_apical_negg": [0.001474043820053339, 0.028314366936683655, 0.08980541676282883], "cos_Ac_negg": [0.44599056243896484, 0.4941726624965668, 0.4544076919555664], "r_norm": [0.19201666116714478, 0.1869010627269745, 0.1835378110408783]}, {"step": 2300, "epoch": 4, "train_loss": 0.12230519950389862, "cos_r_negg": [0.39164015650749207, 0.43627384305000305, 0.4347958564758301], "cos_apical_negg": [-0.0016493774019181728, 0.0343293771147728, 0.10509329289197922], "cos_Ac_negg": [0.44150510430336, 0.48020583391189575, 0.4471698999404907], "r_norm": [0.19779588282108307, 0.19356980919837952, 0.1928449273109436]}, {"epoch_end": 4, "step": 2345, "test_acc": 0.9671, "test_loss": 0.12299122066497803}, {"step": 2400, "epoch": 5, "train_loss": 0.07835916429758072, "cos_r_negg": [0.39116325974464417, 0.44303035736083984, 0.43706852197647095], "cos_apical_negg": [0.0005689701065421104, 0.03678145259618759, 0.10175281018018723], "cos_Ac_negg": [0.43070727586746216, 0.4839155972003937, 0.44670569896698], "r_norm": [0.17620541155338287, 0.17209164798259735, 0.16851550340652466]}, {"step": 2500, "epoch": 5, "train_loss": 0.03489034250378609, "cos_r_negg": [0.3287951350212097, 0.3683970868587494, 0.37697267532348633], "cos_apical_negg": [-0.0003596874885261059, 0.032686635851860046, 0.09533073753118515], "cos_Ac_negg": [0.3681502342224121, 0.4131554365158081, 0.38685232400894165], "r_norm": [0.21968555450439453, 0.21451908349990845, 0.21199902892112732], "ldr": {"dL_sdil": -0.00855495035648346, "dL_bp": -0.009747728705406189, "ratio": 0.8776352538144396}}, {"step": 2600, "epoch": 5, "train_loss": 0.022090446203947067, "cos_r_negg": [0.35706132650375366, 0.39211851358413696, 0.40173202753067017], "cos_apical_negg": [-0.003336722496896982, 0.029070865362882614, 0.09573492407798767], "cos_Ac_negg": [0.3907826244831085, 0.43139955401420593, 0.4079221189022064], "r_norm": [0.1914403736591339, 0.1862468719482422, 0.18351085484027863]}, {"step": 2700, "epoch": 5, "train_loss": 0.04956228286027908, "cos_r_negg": [0.3962959051132202, 0.42606425285339355, 0.42945998907089233], "cos_apical_negg": [-0.006446274928748608, 0.027332115918397903, 0.09695732593536377], "cos_Ac_negg": [0.43585681915283203, 0.4715687036514282, 0.44163092970848083], "r_norm": [0.1681450456380844, 0.16476768255233765, 0.16256943345069885]}, {"step": 2800, "epoch": 5, "train_loss": 0.061803221702575684, "cos_r_negg": [0.3516920804977417, 0.3995523154735565, 0.406349778175354], "cos_apical_negg": [0.010522197932004929, 0.03801455348730087, 0.1026088148355484], "cos_Ac_negg": [0.39506417512893677, 0.44050419330596924, 0.41427141427993774], "r_norm": [0.18657535314559937, 0.18346653878688812, 0.18025299906730652]}, {"epoch_end": 5, "step": 2814, "test_acc": 0.9712, "test_loss": 0.11963936405181885}, {"step": 2900, "epoch": 6, "train_loss": 0.0029091197066009045, "cos_r_negg": [0.3168780207633972, 0.3566356301307678, 0.36284130811691284], "cos_apical_negg": [-0.002426113933324814, 0.03104003146290779, 0.10274471342563629], "cos_Ac_negg": [0.35930225253105164, 0.3978833556175232, 0.37234923243522644], "r_norm": [0.15753984451293945, 0.15396586060523987, 0.15256044268608093]}, {"step": 3000, "epoch": 6, "train_loss": 0.06555234640836716, "cos_r_negg": [0.29954034090042114, 0.3406285345554352, 0.35170841217041016], "cos_apical_negg": [0.004214042332023382, 0.03327927738428116, 0.09258485585451126], "cos_Ac_negg": [0.34664613008499146, 0.3861979842185974, 0.36547720432281494], "r_norm": [0.15212282538414001, 0.14929822087287903, 0.14506179094314575], "ldr": {"dL_sdil": -0.0056327953934669495, "dL_bp": -0.007527798414230347, "ratio": 0.7482659714716676}}, {"step": 3100, "epoch": 6, "train_loss": 0.005165440030395985, "cos_r_negg": [0.2905978262424469, 0.3293302655220032, 0.33835867047309875], "cos_apical_negg": [0.007865103892982006, 0.03524419665336609, 0.0903964713215828], "cos_Ac_negg": [0.3323310315608978, 0.37043750286102295, 0.34681978821754456], "r_norm": [0.1576562076807022, 0.1533031165599823, 0.15176625549793243]}, {"step": 3200, "epoch": 6, "train_loss": 0.024154968559741974, "cos_r_negg": [0.2837607264518738, 0.3160426914691925, 0.3240395188331604], "cos_apical_negg": [0.007447120267897844, 0.02975824847817421, 0.0925460159778595], "cos_Ac_negg": [0.32272806763648987, 0.3581736087799072, 0.33435964584350586], "r_norm": [0.1338508129119873, 0.13185735046863556, 0.12934786081314087]}, {"epoch_end": 6, "step": 3283, "test_acc": 0.972, "test_loss": 0.11885602874755859}, {"step": 3300, "epoch": 7, "train_loss": 0.011400345712900162, "cos_r_negg": [0.2728060185909271, 0.3027855455875397, 0.319587767124176], "cos_apical_negg": [-0.013917852193117142, 0.02657162956893444, 0.10002721101045609], "cos_Ac_negg": [0.31768205761909485, 0.34560680389404297, 0.3265172243118286], "r_norm": [0.1522861123085022, 0.14947503805160522, 0.14627283811569214]}, {"step": 3400, "epoch": 7, "train_loss": 0.01935477927327156, "cos_r_negg": [0.24118351936340332, 0.2715790569782257, 0.2915768623352051], "cos_apical_negg": [-0.00106140598654747, 0.022906463593244553, 0.10799513757228851], "cos_Ac_negg": [0.2775115370750427, 0.3115522265434265, 0.2949219048023224], "r_norm": [0.16454564034938812, 0.16119705140590668, 0.15943534672260284]}, {"step": 3500, "epoch": 7, "train_loss": 0.009277483448386192, "cos_r_negg": [0.22907055914402008, 0.2603045701980591, 0.2829299569129944], "cos_apical_negg": [0.002978227101266384, 0.030053822323679924, 0.1073094829916954], "cos_Ac_negg": [0.2745738625526428, 0.3033802807331085, 0.29376792907714844], "r_norm": [0.14773321151733398, 0.1443905234336853, 0.14236226677894592], "ldr": {"dL_sdil": -0.005328170955181122, "dL_bp": -0.007607020437717438, "ratio": 0.7004281109542927}}, {"step": 3600, "epoch": 7, "train_loss": 0.0019572130404412746, "cos_r_negg": [0.22610728442668915, 0.2564864456653595, 0.2687748372554779], "cos_apical_negg": [-0.0074714538641273975, 0.03298413008451462, 0.1017625704407692], "cos_Ac_negg": [0.2628031075000763, 0.2908572554588318, 0.27389365434646606], "r_norm": [0.16391804814338684, 0.16055110096931458, 0.15846961736679077]}, {"step": 3700, "epoch": 7, "train_loss": 0.006348735652863979, "cos_r_negg": [0.2245323210954666, 0.26149487495422363, 0.27766892313957214], "cos_apical_negg": [-0.003066269913688302, 0.035145848989486694, 0.09773439168930054], "cos_Ac_negg": [0.2678852677345276, 0.30052393674850464, 0.2874765694141388], "r_norm": [0.1471036672592163, 0.14351904392242432, 0.14086958765983582]}, {"epoch_end": 7, "step": 3752, "test_acc": 0.9727, "test_loss": 0.11842634115219117}, {"step": 3800, "epoch": 8, "train_loss": 0.008779313415288925, "cos_r_negg": [0.20920786261558533, 0.24275532364845276, 0.2592512369155884], "cos_apical_negg": [-0.005042918957769871, 0.0364946611225605, 0.10146982222795486], "cos_Ac_negg": [0.25226157903671265, 0.2811870574951172, 0.27063748240470886], "r_norm": [0.15609592199325562, 0.15288537740707397, 0.15019282698631287]}, {"step": 3900, "epoch": 8, "train_loss": 0.029239613562822342, "cos_r_negg": [0.1989031434059143, 0.2301427274942398, 0.24724996089935303], "cos_apical_negg": [-0.00978272408246994, 0.03374071791768074, 0.09561595320701599], "cos_Ac_negg": [0.24267317354679108, 0.2690669596195221, 0.258054256439209], "r_norm": [0.15312577784061432, 0.15000101923942566, 0.1476624608039856]}, {"step": 4000, "epoch": 8, "train_loss": 0.017709914594888687, "cos_r_negg": [0.19030193984508514, 0.2226574569940567, 0.23706385493278503], "cos_apical_negg": [-0.0130704864859581, 0.02808399870991707, 0.09148284792900085], "cos_Ac_negg": [0.2278730422258377, 0.2558026909828186, 0.246942937374115], "r_norm": [0.13309969007968903, 0.13005393743515015, 0.12923449277877808], "ldr": {"dL_sdil": -0.005619361996650696, "dL_bp": -0.0077913179993629456, "ratio": 0.721233813985023}}, {"step": 4100, "epoch": 8, "train_loss": 0.009126199409365654, "cos_r_negg": [0.19683533906936646, 0.2265663743019104, 0.24318739771842957], "cos_apical_negg": [0.0003269389271736145, 0.03210022300481796, 0.08994849026203156], "cos_Ac_negg": [0.2389659285545349, 0.2672910690307617, 0.2573346197605133], "r_norm": [0.13370434939861298, 0.13035769760608673, 0.12766152620315552]}, {"step": 4200, "epoch": 8, "train_loss": 0.016402723267674446, "cos_r_negg": [0.1817178726196289, 0.211750328540802, 0.22712285816669464], "cos_apical_negg": [0.004159575328230858, 0.03204348310828209, 0.09628812223672867], "cos_Ac_negg": [0.22042880952358246, 0.24788206815719604, 0.23804104328155518], "r_norm": [0.15196307003498077, 0.14829201996326447, 0.14516544342041016]}, {"epoch_end": 8, "step": 4221, "test_acc": 0.972, "test_loss": 0.12942726612091066}, {"step": 4300, "epoch": 9, "train_loss": 0.0011772941797971725, "cos_r_negg": [0.171378031373024, 0.2018832415342331, 0.21645718812942505], "cos_apical_negg": [0.0004408275708556175, 0.03153624385595322, 0.09550978988409042], "cos_Ac_negg": [0.21185803413391113, 0.24009981751441956, 0.2312624454498291], "r_norm": [0.16429108381271362, 0.16050873696804047, 0.1585191786289215]}, {"step": 4400, "epoch": 9, "train_loss": 0.0017589031485840678, "cos_r_negg": [0.18395528197288513, 0.21181973814964294, 0.22694475948810577], "cos_apical_negg": [0.004927505739033222, 0.03397145867347717, 0.09346376359462738], "cos_Ac_negg": [0.22359634935855865, 0.25286808609962463, 0.2441624402999878], "r_norm": [0.15238656103610992, 0.14788872003555298, 0.1455930471420288]}, {"step": 4500, "epoch": 9, "train_loss": 0.0499572828412056, "cos_r_negg": [0.16712220013141632, 0.19889728724956512, 0.21286870539188385], "cos_apical_negg": [0.000697003211826086, 0.03215166926383972, 0.0884338840842247], "cos_Ac_negg": [0.20319169759750366, 0.227967768907547, 0.22162587940692902], "r_norm": [0.1630411297082901, 0.15995687246322632, 0.15871787071228027], "ldr": {"dL_sdil": -0.007485546171665192, "dL_bp": -0.011915862560272217, "ratio": 0.6282001100467699}}, {"step": 4600, "epoch": 9, "train_loss": 0.00047172076301649213, "cos_r_negg": [0.1720544397830963, 0.19713377952575684, 0.20793026685714722], "cos_apical_negg": [0.0006287680007517338, 0.03027612343430519, 0.08023521304130554], "cos_Ac_negg": [0.2048678696155548, 0.22890585660934448, 0.2191743701696396], "r_norm": [0.14321473240852356, 0.13957840204238892, 0.13830171525478363]}, {"epoch_end": 9, "step": 4690, "test_acc": 0.9717, "test_loss": 0.1321223569869995}, {"step": 4700, "epoch": 10, "train_loss": 0.002440714742988348, "cos_r_negg": [0.15346384048461914, 0.17721645534038544, 0.19002754986286163], "cos_apical_negg": [-0.007044458761811256, 0.030264757573604584, 0.0846354067325592], "cos_Ac_negg": [0.18284618854522705, 0.20435011386871338, 0.1971147656440735], "r_norm": [0.15336430072784424, 0.14960631728172302, 0.14814691245555878]}, {"step": 4800, "epoch": 10, "train_loss": 0.004119536839425564, "cos_r_negg": [0.14772538840770721, 0.17196312546730042, 0.18439564108848572], "cos_apical_negg": [-0.00809923093765974, 0.030700212344527245, 0.08027894794940948], "cos_Ac_negg": [0.17864659428596497, 0.19959218800067902, 0.19346624612808228], "r_norm": [0.1517089605331421, 0.14809250831604004, 0.14698970317840576]}, {"step": 4900, "epoch": 10, "train_loss": 0.0019432123517617583, "cos_r_negg": [0.1436895728111267, 0.16759449243545532, 0.18327967822551727], "cos_apical_negg": [-0.004903081804513931, 0.029023364186286926, 0.08085942268371582], "cos_Ac_negg": [0.17696651816368103, 0.19628387689590454, 0.19116395711898804], "r_norm": [0.1441417783498764, 0.14046290516853333, 0.13939125835895538]}, {"step": 5000, "epoch": 10, "train_loss": 0.0011039283126592636, "cos_r_negg": [0.1456233710050583, 0.16664552688598633, 0.18366661667823792], "cos_apical_negg": [-0.00496734818443656, 0.027270881459116936, 0.07906773686408997], "cos_Ac_negg": [0.177052304148674, 0.19654516875743866, 0.192416250705719], "r_norm": [0.153082475066185, 0.1500624716281891, 0.1487290859222412], "ldr": {"dL_sdil": -0.00773930549621582, "dL_bp": -0.012882724404335022, "ratio": 0.6007506838857434}}, {"step": 5100, "epoch": 10, "train_loss": 0.0019905571825802326, "cos_r_negg": [0.14017845690250397, 0.1623867303133011, 0.1790478378534317], "cos_apical_negg": [-0.006340588442981243, 0.02820134535431862, 0.07454545050859451], "cos_Ac_negg": [0.17037701606750488, 0.18867020308971405, 0.18366976082324982], "r_norm": [0.1452169120311737, 0.1420271098613739, 0.14085450768470764]}, {"epoch_end": 10, "step": 5159, "test_acc": 0.9727, "test_loss": 0.13825255422592164}, {"step": 5200, "epoch": 11, "train_loss": 0.00032047624699771404, "cos_r_negg": [0.13907811045646667, 0.16218888759613037, 0.17825764417648315], "cos_apical_negg": [-0.004437034949660301, 0.027674730867147446, 0.07651957124471664], "cos_Ac_negg": [0.16903731226921082, 0.18682733178138733, 0.18353214859962463], "r_norm": [0.14994730055332184, 0.14747193455696106, 0.1453215479850769]}, {"step": 5300, "epoch": 11, "train_loss": 0.001317075453698635, "cos_r_negg": [0.1397765725851059, 0.15966805815696716, 0.17645254731178284], "cos_apical_negg": [-0.007834138348698616, 0.027934767305850983, 0.0747075229883194], "cos_Ac_negg": [0.17215052247047424, 0.19023580849170685, 0.185514435172081], "r_norm": [0.15098218619823456, 0.14723503589630127, 0.14609037339687347]}, {"step": 5400, "epoch": 11, "train_loss": 0.0019159776857122779, "cos_r_negg": [0.13290445506572723, 0.1524316817522049, 0.16961708664894104], "cos_apical_negg": [-0.011777791194617748, 0.027024639770388603, 0.07163079082965851], "cos_Ac_negg": [0.16187609732151031, 0.1794547736644745, 0.1759195327758789], "r_norm": [0.14193996787071228, 0.13908618688583374, 0.1379406601190567]}, {"step": 5500, "epoch": 11, "train_loss": 0.0018864399753510952, "cos_r_negg": [0.1298910677433014, 0.15068814158439636, 0.16763979196548462], "cos_apical_negg": [-0.01511571928858757, 0.02759353071451187, 0.07189469039440155], "cos_Ac_negg": [0.15988746285438538, 0.17757219076156616, 0.17409983277320862], "r_norm": [0.14620211720466614, 0.14322549104690552, 0.14220669865608215], "ldr": {"dL_sdil": -0.007692717015743256, "dL_bp": -0.01239149272441864, "ratio": 0.6208063214679543}}, {"step": 5600, "epoch": 11, "train_loss": 0.00045360150397755206, "cos_r_negg": [0.12779346108436584, 0.14734984934329987, 0.1627187728881836], "cos_apical_negg": [-0.0132527407258749, 0.02890801802277565, 0.07118037343025208], "cos_Ac_negg": [0.15754546225070953, 0.17433032393455505, 0.1712392419576645], "r_norm": [0.1398567259311676, 0.1368670016527176, 0.1359477937221527]}, {"epoch_end": 11, "step": 5628, "test_acc": 0.9727, "test_loss": 0.1380262689590454}, {"step": 5700, "epoch": 12, "train_loss": 0.0010436055017635226, "cos_r_negg": [0.12593592703342438, 0.14469364285469055, 0.16063137352466583], "cos_apical_negg": [-0.014148636721074581, 0.02715049311518669, 0.06879954785108566], "cos_Ac_negg": [0.15484890341758728, 0.17134228348731995, 0.16855867207050323], "r_norm": [0.13897186517715454, 0.13628479838371277, 0.1349405199289322]}, {"step": 5800, "epoch": 12, "train_loss": 0.000346480926964432, "cos_r_negg": [0.12032702565193176, 0.13968771696090698, 0.15657800436019897], "cos_apical_negg": [-0.013153620064258575, 0.026260606944561005, 0.06714522838592529], "cos_Ac_negg": [0.15047548711299896, 0.1682988554239273, 0.1652776002883911], "r_norm": [0.14279009401798248, 0.13989438116550446, 0.1387310028076172]}, {"step": 5900, "epoch": 12, "train_loss": 0.001038936898112297, "cos_r_negg": [0.12642058730125427, 0.1462206393480301, 0.1628953218460083], "cos_apical_negg": [-0.014939341694116592, 0.027533335611224174, 0.06816059350967407], "cos_Ac_negg": [0.15559664368629456, 0.17304499447345734, 0.17043958604335785], "r_norm": [0.14304625988006592, 0.1399763524532318, 0.13858327269554138]}, {"step": 6000, "epoch": 12, "train_loss": 0.0006938993465155363, "cos_r_negg": [0.12021420150995255, 0.13889288902282715, 0.15497863292694092], "cos_apical_negg": [-0.01536024920642376, 0.027249818667769432, 0.06561246514320374], "cos_Ac_negg": [0.14953388273715973, 0.16581204533576965, 0.16294600069522858], "r_norm": [0.14249981939792633, 0.13942919671535492, 0.13825908303260803], "ldr": {"dL_sdil": -0.0077394768595695496, "dL_bp": -0.013016998767852783, "ratio": 0.5945669195792829}}, {"epoch_end": 12, "step": 6097, "test_acc": 0.9728, "test_loss": 0.14158571338653564}, {"step": 6100, "epoch": 13, "train_loss": 0.0006675304030068219, "cos_r_negg": [0.12018933892250061, 0.13733035326004028, 0.15323753654956818], "cos_apical_negg": [-0.01626214012503624, 0.02698269486427307, 0.06725136935710907], "cos_Ac_negg": [0.14689643681049347, 0.16226354241371155, 0.15944406390190125], "r_norm": [0.14575155079364777, 0.1428396850824356, 0.1414227932691574]}, {"step": 6200, "epoch": 13, "train_loss": 0.0004157752846367657, "cos_r_negg": [0.11543910205364227, 0.1320739984512329, 0.14808131754398346], "cos_apical_negg": [-0.016483131796121597, 0.02685311622917652, 0.06537206470966339], "cos_Ac_negg": [0.1437090039253235, 0.15889741480350494, 0.15584781765937805], "r_norm": [0.14185965061187744, 0.1389819085597992, 0.13789618015289307]}, {"step": 6300, "epoch": 13, "train_loss": 0.0011423379182815552, "cos_r_negg": [0.1137993186712265, 0.1301238238811493, 0.14570337533950806], "cos_apical_negg": [-0.017200684174895287, 0.026560001075267792, 0.06553532183170319], "cos_Ac_negg": [0.14120899140834808, 0.15593808889389038, 0.15304967761039734], "r_norm": [0.14338231086730957, 0.14059233665466309, 0.1393817961215973]}, {"step": 6400, "epoch": 13, "train_loss": 0.0003152909630443901, "cos_r_negg": [0.11512938141822815, 0.13242846727371216, 0.14829787611961365], "cos_apical_negg": [-0.015821989625692368, 0.02621270716190338, 0.06602505594491959], "cos_Ac_negg": [0.14223501086235046, 0.15746155381202698, 0.15523487329483032], "r_norm": [0.1417124718427658, 0.13895007967948914, 0.13768382370471954]}, {"step": 6500, "epoch": 13, "train_loss": 0.00015983209596015513, "cos_r_negg": [0.11377174407243729, 0.1315324753522873, 0.1473967730998993], "cos_apical_negg": [-0.01693193055689335, 0.02656492590904236, 0.06760591268539429], "cos_Ac_negg": [0.14286907017230988, 0.15832945704460144, 0.15576104819774628], "r_norm": [0.14350740611553192, 0.14046916365623474, 0.1392725259065628], "ldr": {"dL_sdil": -0.00824255496263504, "dL_bp": -0.014194592833518982, "ratio": 0.5806827331581605}}, {"epoch_end": 13, "step": 6566, "test_acc": 0.9727, "test_loss": 0.14322918624877928}, {"step": 6600, "epoch": 14, "train_loss": 0.0003591316635720432, "cos_r_negg": [0.11212986707687378, 0.1291949599981308, 0.14487352967262268], "cos_apical_negg": [-0.014751208014786243, 0.02600904554128647, 0.06539197266101837], "cos_Ac_negg": [0.1404675394296646, 0.15541696548461914, 0.15315845608711243], "r_norm": [0.13971495628356934, 0.13707971572875977, 0.13579395413398743]}, {"step": 6700, "epoch": 14, "train_loss": 0.0004953234456479549, "cos_r_negg": [0.11256467550992966, 0.12910374999046326, 0.1448487937450409], "cos_apical_negg": [-0.014729049056768417, 0.025265991687774658, 0.0659550279378891], "cos_Ac_negg": [0.14007018506526947, 0.155273899435997, 0.15296408534049988], "r_norm": [0.1439666450023651, 0.14105452597141266, 0.13976356387138367]}, {"step": 6800, "epoch": 14, "train_loss": 0.0005221293540671468, "cos_r_negg": [0.1103338971734047, 0.12624657154083252, 0.1417093127965927], "cos_apical_negg": [-0.015459051355719566, 0.025464942678809166, 0.06509223580360413], "cos_Ac_negg": [0.13784286379814148, 0.15203189849853516, 0.15006974339485168], "r_norm": [0.1405353844165802, 0.13785463571548462, 0.13657337427139282]}, {"step": 6900, "epoch": 14, "train_loss": 0.0006800569244660437, "cos_r_negg": [0.10567227005958557, 0.12139198929071426, 0.13738387823104858], "cos_apical_negg": [-0.017395276576280594, 0.025409504771232605, 0.064261794090271], "cos_Ac_negg": [0.1346426159143448, 0.14878396689891815, 0.14626097679138184], "r_norm": [0.13921457529067993, 0.13651230931282043, 0.13532032072544098]}, {"step": 7000, "epoch": 14, "train_loss": 0.000475144712254405, "cos_r_negg": [0.1054995208978653, 0.12289255857467651, 0.13774460554122925], "cos_apical_negg": [-0.01858312450349331, 0.026010826230049133, 0.06366252899169922], "cos_Ac_negg": [0.13563184440135956, 0.1502842903137207, 0.14762620627880096], "r_norm": [0.14071890711784363, 0.13789233565330505, 0.1369301676750183], "ldr": {"dL_sdil": -0.008366659283638, "dL_bp": -0.014673717319965363, "ratio": 0.57017994153766}}, {"epoch_end": 14, "step": 7035, "test_acc": 0.9731, "test_loss": 0.14614157485961915}], "final": {"test_acc": 0.9731, "test_loss": 0.14614157485961915, "wall_s": 41.07883429527283, "cos_r_negg": [0.10460954904556274, 0.12169840931892395, 0.13700059056282043], "cos_apical_negg": [-0.01877596229314804, 0.026302088052034378, 0.06400805711746216], "cos_Ac_negg": [0.13483735918998718, 0.14949391782283783, 0.14689740538597107]}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_bp_w64_d30_s0.json b/results/pilot_jac_cifar10_bp_w64_d30_s0.json new file mode 100644 index 0000000..e5bc9f8 --- /dev/null +++ b/results/pilot_jac_cifar10_bp_w64_d30_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "bp", "dataset": "cifar10", "depth": 30, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_bp_w64_d30_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.415728807449341}, {"epoch_end": 0, "step": 391, "test_acc": 0.3648, "test_loss": 1.8013721069335937}, {"step": 500, "epoch": 1, "train_loss": 1.6938464641571045}, {"epoch_end": 1, "step": 782, "test_acc": 0.3924, "test_loss": 1.7378808959960939}, {"step": 1000, "epoch": 2, "train_loss": 1.7924134731292725}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4074, "test_loss": 1.7012336791992186}, {"step": 1500, "epoch": 3, "train_loss": 1.6040619611740112}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4164, "test_loss": 1.68049423828125}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4131, "test_loss": 1.6608419555664062}], "final": {"test_acc": 0.4131, "test_loss": 1.6608419555664062, "wall_s": 29.352590799331665}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_bp_w64_d60_s0.json b/results/pilot_jac_cifar10_bp_w64_d60_s0.json new file mode 100644 index 0000000..f3152d1 --- /dev/null +++ b/results/pilot_jac_cifar10_bp_w64_d60_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "bp", "dataset": "cifar10", "depth": 60, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_bp_w64_d60_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.6154472827911377}, {"epoch_end": 0, "step": 391, "test_acc": 0.3756, "test_loss": 1.7967720581054687}, {"step": 500, "epoch": 1, "train_loss": 1.764350414276123}, {"epoch_end": 1, "step": 782, "test_acc": 0.3947, "test_loss": 1.7364661865234374}, {"step": 1000, "epoch": 2, "train_loss": 1.71304452419281}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4056, "test_loss": 1.7077377685546875}, {"step": 1500, "epoch": 3, "train_loss": 1.5759260654449463}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4126, "test_loss": 1.6889246704101561}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4293, "test_loss": 1.6495174194335938}], "final": {"test_acc": 0.4293, "test_loss": 1.6495174194335938, "wall_s": 63.739113569259644}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_dfa_w64_d30_s0.json b/results/pilot_jac_cifar10_dfa_w64_d30_s0.json new file mode 100644 index 0000000..6cb0e96 --- /dev/null +++ b/results/pilot_jac_cifar10_dfa_w64_d30_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "dfa", "dataset": "cifar10", "depth": 30, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_dfa_w64_d30_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.4157285690307617, "cos_r_negg": [0.04130179435014725, -0.05167870968580246, 0.029879383742809296, 0.03635120019316673, -0.00517943175509572, -0.0037146983668208122, 0.014719751663506031, -0.012813427485525608, 0.08911487460136414, -0.025006676092743874, -0.043925944715738297, 0.004191750194877386, 0.0028503923676908016, 0.03278622776269913, -0.023855693638324738, 0.03571062535047531, -0.015357001684606075, -2.7436763048171997e-05, 0.010302920825779438, 0.02555227465927601, 0.040136873722076416, -0.010950050316751003, 0.03490540385246277, -0.029108267277479172, -0.04761591553688049, -0.03802322596311569, 0.015627088025212288, 0.0047096796333789825, 0.055373527109622955, 0.043867871165275574], "cos_apical_negg": [0.04130179435014725, -0.05167870968580246, 0.029879383742809296, 0.03635120019316673, -0.00517943175509572, -0.0037146983668208122, 0.014719751663506031, -0.012813427485525608, 0.08911487460136414, -0.025006676092743874, -0.043925944715738297, 0.004191750194877386, 0.0028503923676908016, 0.03278622776269913, -0.023855693638324738, 0.03571062535047531, -0.015357001684606075, -2.7436763048171997e-05, 0.010302920825779438, 0.02555227465927601, 0.040136873722076416, -0.010950050316751003, 0.03490540385246277, -0.029108267277479172, -0.04761591553688049, -0.03802322596311569, 0.015627088025212288, 0.0047096796333789825, 0.055373527109622955, 0.043867871165275574], "cos_Ac_negg": [0.04130179435014725, -0.05167870968580246, 0.029879383742809296, 0.03635120019316673, -0.00517943175509572, -0.0037146983668208122, 0.014719751663506031, -0.012813427485525608, 0.08911487460136414, -0.025006676092743874, -0.043925944715738297, 0.004191750194877386, 0.0028503923676908016, 0.03278622776269913, -0.023855693638324738, 0.03571062535047531, -0.015357001684606075, -2.7436763048171997e-05, 0.010302920825779438, 0.02555227465927601, 0.040136873722076416, -0.010950050316751003, 0.03490540385246277, -0.029108267277479172, -0.04761591553688049, -0.03802322596311569, 0.015627088025212288, 0.0047096796333789825, 0.055373527109622955, 0.043867871165275574], "r_norm": [2.6118388175964355, 2.5107076168060303, 2.381376266479492, 2.5533318519592285, 2.4093616008758545, 2.4340720176696777, 2.5209712982177734, 2.47703218460083, 2.3975400924682617, 2.395562171936035, 2.4695844650268555, 2.3991799354553223, 2.5008273124694824, 2.5226407051086426, 2.46388578414917, 2.44757080078125, 2.4495849609375, 2.4044461250305176, 2.402073860168457, 2.440375566482544, 2.5851316452026367, 2.459933280944824, 2.5742692947387695, 2.475496292114258, 2.421234607696533, 2.4259753227233887, 2.365795135498047, 2.4156174659729004, 2.435579299926758, 2.5802385807037354]}, {"epoch_end": 0, "step": 391, "test_acc": 0.3439, "test_loss": 1.8688878173828125}, {"step": 500, "epoch": 1, "train_loss": 1.8185863494873047, "cos_r_negg": [0.5021442174911499, -0.024272777140140533, -0.010427339933812618, 0.028790874406695366, -0.007071430329233408, 0.041475072503089905, -0.06685444712638855, -0.0792425200343132, 0.03061535768210888, 0.013030911795794964, -0.013616947457194328, -0.005817234516143799, -0.05089216306805611, -0.022574737668037415, 0.00764061463996768, 0.008589653298258781, -0.08776582777500153, 0.027161289006471634, 0.008954466320574284, -0.01848287507891655, -0.03218787536025047, -0.013208515010774136, 0.016982974484562874, 0.11147912591695786, -0.0019691158086061478, 0.06348760426044464, 0.05869927257299423, 0.008566780015826225, 0.0175030454993248, -0.015016330406069756], "cos_apical_negg": [0.5021442174911499, -0.024272777140140533, -0.010427339933812618, 0.028790874406695366, -0.007071430329233408, 0.041475072503089905, -0.06685444712638855, -0.0792425200343132, 0.03061535768210888, 0.013030911795794964, -0.013616947457194328, -0.005817234516143799, -0.05089216306805611, -0.022574737668037415, 0.00764061463996768, 0.008589653298258781, -0.08776582777500153, 0.027161289006471634, 0.008954466320574284, -0.01848287507891655, -0.03218787536025047, -0.013208515010774136, 0.016982974484562874, 0.11147912591695786, -0.0019691158086061478, 0.06348760426044464, 0.05869927257299423, 0.008566780015826225, 0.0175030454993248, -0.015016330406069756], "cos_Ac_negg": [0.5021442174911499, -0.024272777140140533, -0.010427339933812618, 0.028790874406695366, -0.007071430329233408, 0.041475072503089905, -0.06685444712638855, -0.0792425200343132, 0.03061535768210888, 0.013030911795794964, -0.013616947457194328, -0.005817234516143799, -0.05089216306805611, -0.022574737668037415, 0.00764061463996768, 0.008589653298258781, -0.08776582777500153, 0.027161289006471634, 0.008954466320574284, -0.01848287507891655, -0.03218787536025047, -0.013208515010774136, 0.016982974484562874, 0.11147912591695786, -0.0019691158086061478, 0.06348760426044464, 0.05869927257299423, 0.008566780015826225, 0.0175030454993248, -0.015016330406069756], "r_norm": [2.2987172603607178, 2.2241992950439453, 2.116129159927368, 2.276242733001709, 2.134511947631836, 2.147556781768799, 2.2263612747192383, 2.191457509994507, 2.1340630054473877, 2.1296725273132324, 2.2165956497192383, 2.130927324295044, 2.2357640266418457, 2.2310521602630615, 2.1970715522766113, 2.1958048343658447, 2.178600788116455, 2.134237766265869, 2.1169447898864746, 2.1933095455169678, 2.276766777038574, 2.211942195892334, 2.2697978019714355, 2.16856050491333, 2.185072183609009, 2.142336368560791, 2.078704357147217, 2.170818567276001, 2.163302421569824, 2.2897000312805176]}, {"epoch_end": 1, "step": 782, "test_acc": 0.357, "test_loss": 1.8430688598632812}, {"step": 1000, "epoch": 2, "train_loss": 1.8754260540008545, "cos_r_negg": [0.5487514734268188, 0.007248611655086279, -0.008624782785773277, -0.0057631745003163815, -0.031145749613642693, 0.052426233887672424, -0.044822391122579575, -0.07710064947605133, 0.020750969648361206, 0.026115912944078445, -0.02820703759789467, 0.016865402460098267, 0.0038375824224203825, -0.024961546063423157, 0.01999833434820175, -0.02775469422340393, -0.06761563569307327, 0.04648683965206146, 0.01960933953523636, -0.0023083328269422054, -0.06000566482543945, -0.05465877056121826, 0.016128091141581535, 0.07718849182128906, 0.01232006773352623, 0.06577295064926147, 0.07419411838054657, -0.017742210999131203, 0.01880396343767643, 0.05452614277601242], "cos_apical_negg": [0.5487514734268188, 0.007248611655086279, -0.008624782785773277, -0.0057631745003163815, -0.031145749613642693, 0.052426233887672424, -0.044822391122579575, -0.07710064947605133, 0.020750969648361206, 0.026115912944078445, -0.02820703759789467, 0.016865402460098267, 0.0038375824224203825, -0.024961546063423157, 0.01999833434820175, -0.02775469422340393, -0.06761563569307327, 0.04648683965206146, 0.01960933953523636, -0.0023083328269422054, -0.06000566482543945, -0.05465877056121826, 0.016128091141581535, 0.07718849182128906, 0.01232006773352623, 0.06577295064926147, 0.07419411838054657, -0.017742210999131203, 0.01880396343767643, 0.05452614277601242], "cos_Ac_negg": [0.5487514734268188, 0.007248611655086279, -0.008624782785773277, -0.0057631745003163815, -0.031145749613642693, 0.052426233887672424, -0.044822391122579575, -0.07710064947605133, 0.020750969648361206, 0.026115912944078445, -0.02820703759789467, 0.016865402460098267, 0.0038375824224203825, -0.024961546063423157, 0.01999833434820175, -0.02775469422340393, -0.06761563569307327, 0.04648683965206146, 0.01960933953523636, -0.0023083328269422054, -0.06000566482543945, -0.05465877056121826, 0.016128091141581535, 0.07718849182128906, 0.01232006773352623, 0.06577295064926147, 0.07419411838054657, -0.017742210999131203, 0.01880396343767643, 0.05452614277601242], "r_norm": [2.3114936351776123, 2.2366087436676025, 2.122636079788208, 2.288330316543579, 2.1441550254821777, 2.16719388961792, 2.238264560699463, 2.2185521125793457, 2.143141984939575, 2.1465981006622314, 2.2425267696380615, 2.140319347381592, 2.2467145919799805, 2.2544517517089844, 2.2014012336730957, 2.210102081298828, 2.202552318572998, 2.152041435241699, 2.1359610557556152, 2.2039899826049805, 2.271368980407715, 2.221750020980835, 2.2761611938476562, 2.179783344268799, 2.194851875305176, 2.147956609725952, 2.0958988666534424, 2.188201904296875, 2.171701192855835, 2.3094558715820312]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.3618, "test_loss": 1.8217102172851563}, {"step": 1500, "epoch": 3, "train_loss": 1.7128980159759521, "cos_r_negg": [0.5362775921821594, 0.011473788879811764, -0.011900755576789379, 0.02257465198636055, 0.021306442096829414, 0.04985255002975464, -0.02962353453040123, -0.019608264788985252, 0.03569295257329941, 0.07186584174633026, -0.007914965972304344, 0.008661587722599506, -0.053245604038238525, 0.08209188282489777, 0.05064284801483154, 0.0038046101108193398, 0.021014947444200516, 0.08605606853961945, 0.014818879775702953, 0.03713211417198181, -0.0714733898639679, 0.02223656326532364, 0.05603468045592308, 0.06525898724794388, 0.021623045206069946, 0.058812737464904785, 0.08270993828773499, 0.028872953727841377, 0.009778222069144249, 0.0332469716668129], "cos_apical_negg": [0.5362775921821594, 0.011473788879811764, -0.011900755576789379, 0.02257465198636055, 0.021306442096829414, 0.04985255002975464, -0.02962353453040123, -0.019608264788985252, 0.03569295257329941, 0.07186584174633026, -0.007914965972304344, 0.008661587722599506, -0.053245604038238525, 0.08209188282489777, 0.05064284801483154, 0.0038046101108193398, 0.021014947444200516, 0.08605606853961945, 0.014818879775702953, 0.03713211417198181, -0.0714733898639679, 0.02223656326532364, 0.05603468045592308, 0.06525898724794388, 0.021623045206069946, 0.058812737464904785, 0.08270993828773499, 0.028872953727841377, 0.009778222069144249, 0.0332469716668129], "cos_Ac_negg": [0.5362775921821594, 0.011473788879811764, -0.011900755576789379, 0.02257465198636055, 0.021306442096829414, 0.04985255002975464, -0.02962353453040123, -0.019608264788985252, 0.03569295257329941, 0.07186584174633026, -0.007914965972304344, 0.008661587722599506, -0.053245604038238525, 0.08209188282489777, 0.05064284801483154, 0.0038046101108193398, 0.021014947444200516, 0.08605606853961945, 0.014818879775702953, 0.03713211417198181, -0.0714733898639679, 0.02223656326532364, 0.05603468045592308, 0.06525898724794388, 0.021623045206069946, 0.058812737464904785, 0.08270993828773499, 0.028872953727841377, 0.009778222069144249, 0.0332469716668129], "r_norm": [2.2674660682678223, 2.2026212215423584, 2.090864658355713, 2.2454700469970703, 2.1055290699005127, 2.118560791015625, 2.190725326538086, 2.1559906005859375, 2.1015584468841553, 2.0963563919067383, 2.183671712875366, 2.101579427719116, 2.1998825073242188, 2.208488702774048, 2.1671369075775146, 2.1577255725860596, 2.1452183723449707, 2.109074831008911, 2.090406656265259, 2.1647396087646484, 2.2389886379241943, 2.178550958633423, 2.2406816482543945, 2.1366395950317383, 2.152134895324707, 2.110382556915283, 2.0520753860473633, 2.142508029937744, 2.135183334350586, 2.253416061401367]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.3724, "test_loss": 1.81636044921875}, {"epoch_end": 4, "step": 1955, "test_acc": 0.3731, "test_loss": 1.8001628662109375}], "final": {"test_acc": 0.3731, "test_loss": 1.8001628662109375, "wall_s": 19.448094367980957, "cos_r_negg": [0.6099982261657715, 0.041578128933906555, -0.006816087290644646, 0.03268715739250183, 0.020214516669511795, 0.04716081917285919, -0.008767107501626015, -0.040753573179244995, 0.05464443564414978, 0.09139872342348099, -0.003479434410110116, 0.03690046817064285, -0.019361397251486778, 0.0345904566347599, 0.018553491681814194, 0.07845322042703629, -0.011157560162246227, 0.0830966979265213, 0.08898749947547913, 0.009537972509860992, -0.009123572148382664, -0.004601207561790943, 0.028817608952522278, 0.07400146126747131, 0.0473867766559124, 0.11561077833175659, 0.06366584450006485, -0.02335822954773903, 0.06951933354139328, 0.019080501049757004], "cos_apical_negg": [0.6099982261657715, 0.041578128933906555, -0.006816087290644646, 0.03268715739250183, 0.020214516669511795, 0.04716081917285919, -0.008767107501626015, -0.040753573179244995, 0.05464443564414978, 0.09139872342348099, -0.003479434410110116, 0.03690046817064285, -0.019361397251486778, 0.0345904566347599, 0.018553491681814194, 0.07845322042703629, -0.011157560162246227, 0.0830966979265213, 0.08898749947547913, 0.009537972509860992, -0.009123572148382664, -0.004601207561790943, 0.028817608952522278, 0.07400146126747131, 0.0473867766559124, 0.11561077833175659, 0.06366584450006485, -0.02335822954773903, 0.06951933354139328, 0.019080501049757004], "cos_Ac_negg": [0.6099982261657715, 0.041578128933906555, -0.006816087290644646, 0.03268715739250183, 0.020214516669511795, 0.04716081917285919, -0.008767107501626015, -0.040753573179244995, 0.05464443564414978, 0.09139872342348099, -0.003479434410110116, 0.03690046817064285, -0.019361397251486778, 0.0345904566347599, 0.018553491681814194, 0.07845322042703629, -0.011157560162246227, 0.0830966979265213, 0.08898749947547913, 0.009537972509860992, -0.009123572148382664, -0.004601207561790943, 0.028817608952522278, 0.07400146126747131, 0.0473867766559124, 0.11561077833175659, 0.06366584450006485, -0.02335822954773903, 0.06951933354139328, 0.019080501049757004]}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_dfa_w64_d60_s0.json b/results/pilot_jac_cifar10_dfa_w64_d60_s0.json new file mode 100644 index 0000000..7e78abd --- /dev/null +++ b/results/pilot_jac_cifar10_dfa_w64_d60_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "dfa", "dataset": "cifar10", "depth": 60, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.05, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_dfa_w64_d60_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.6154470443725586, "cos_r_negg": [-0.006716079078614712, 0.02438841015100479, -0.04525028541684151, 0.013871347531676292, -0.006345417816191912, -0.020512912422418594, -0.06818285584449768, -0.03158885985612869, -0.022064300253987312, -0.05176926404237747, 0.03123473934829235, -0.0071239229291677475, 0.0023502311669290066, -0.06361411511898041, -0.06265509873628616, -0.0626712515950203, 0.07430143654346466, 0.06538615375757217, 0.07140052318572998, -0.029584277421236038, 0.026476535946130753, -0.013373231515288353, 0.06404337286949158, 0.01492207683622837, -0.001631920924410224, -0.004349656403064728, -0.07081231474876404, -0.010206866078078747, -0.054892025887966156, -0.024155914783477783, 0.039311427623033524, -0.0015584426000714302, -0.028794849291443825, -0.028474921360611916, -0.008811524137854576, 0.002567646559327841, -0.002357651712372899, -0.013387875631451607, 0.034821949899196625, 0.011808709241449833, 0.10773257911205292, 0.027077380567789078, -0.02375342883169651, 0.04900947958230972, -0.033205289393663406, 0.003592877183109522, -0.045937854796648026, 0.0025787833146750927, -0.06160680949687958, 0.062074556946754456, -0.0015370987821370363, -0.09670576453208923, 0.05235959589481354, 0.026170790195465088, -0.02327905222773552, 0.05072954297065735, -0.0032390174455940723, -0.07727590948343277, -0.04464735835790634, -0.03091677464544773], "cos_apical_negg": [-0.006716079078614712, 0.02438841015100479, -0.04525028541684151, 0.013871347531676292, -0.006345417816191912, -0.020512912422418594, -0.06818285584449768, -0.03158885985612869, -0.022064300253987312, -0.05176926404237747, 0.03123473934829235, -0.0071239229291677475, 0.0023502311669290066, -0.06361411511898041, -0.06265509873628616, -0.0626712515950203, 0.07430143654346466, 0.06538615375757217, 0.07140052318572998, -0.029584277421236038, 0.026476535946130753, -0.013373231515288353, 0.06404337286949158, 0.01492207683622837, -0.001631920924410224, -0.004349656403064728, -0.07081231474876404, -0.010206866078078747, -0.054892025887966156, -0.024155914783477783, 0.039311427623033524, -0.0015584426000714302, -0.028794849291443825, -0.028474921360611916, -0.008811524137854576, 0.002567646559327841, -0.002357651712372899, -0.013387875631451607, 0.034821949899196625, 0.011808709241449833, 0.10773257911205292, 0.027077380567789078, -0.02375342883169651, 0.04900947958230972, -0.033205289393663406, 0.003592877183109522, -0.045937854796648026, 0.0025787833146750927, -0.06160680949687958, 0.062074556946754456, -0.0015370987821370363, -0.09670576453208923, 0.05235959589481354, 0.026170790195465088, -0.02327905222773552, 0.05072954297065735, -0.0032390174455940723, -0.07727590948343277, -0.04464735835790634, -0.03091677464544773], "cos_Ac_negg": [-0.006716079078614712, 0.02438841015100479, -0.04525028541684151, 0.013871347531676292, -0.006345417816191912, -0.020512912422418594, -0.06818285584449768, -0.03158885985612869, -0.022064300253987312, -0.05176926404237747, 0.03123473934829235, -0.0071239229291677475, 0.0023502311669290066, -0.06361411511898041, -0.06265509873628616, -0.0626712515950203, 0.07430143654346466, 0.06538615375757217, 0.07140052318572998, -0.029584277421236038, 0.026476535946130753, -0.013373231515288353, 0.06404337286949158, 0.01492207683622837, -0.001631920924410224, -0.004349656403064728, -0.07081231474876404, -0.010206866078078747, -0.054892025887966156, -0.024155914783477783, 0.039311427623033524, -0.0015584426000714302, -0.028794849291443825, -0.028474921360611916, -0.008811524137854576, 0.002567646559327841, -0.002357651712372899, -0.013387875631451607, 0.034821949899196625, 0.011808709241449833, 0.10773257911205292, 0.027077380567789078, -0.02375342883169651, 0.04900947958230972, -0.033205289393663406, 0.003592877183109522, -0.045937854796648026, 0.0025787833146750927, -0.06160680949687958, 0.062074556946754456, -0.0015370987821370363, -0.09670576453208923, 0.05235959589481354, 0.026170790195465088, -0.02327905222773552, 0.05072954297065735, -0.0032390174455940723, -0.07727590948343277, -0.04464735835790634, -0.03091677464544773], "r_norm": [2.476837158203125, 2.588827610015869, 2.507465362548828, 2.4364213943481445, 2.284212112426758, 2.463568687438965, 2.3808512687683105, 2.447174072265625, 2.5927276611328125, 2.5126051902770996, 2.4167494773864746, 2.584993839263916, 2.4760541915893555, 2.4647979736328125, 2.611776351928711, 2.5096240043640137, 2.580117702484131, 2.5247292518615723, 2.476726531982422, 2.5275793075561523, 2.617313861846924, 2.4330687522888184, 2.438448429107666, 2.5159451961517334, 2.457703113555908, 2.434385299682617, 2.6349055767059326, 2.4479267597198486, 2.3842978477478027, 2.6444690227508545, 2.3573412895202637, 2.4539451599121094, 2.4437050819396973, 2.4726710319519043, 2.5920939445495605, 2.5337939262390137, 2.3910603523254395, 2.3863282203674316, 2.5472004413604736, 2.4595766067504883, 2.525862455368042, 2.42592191696167, 2.493037223815918, 2.4348723888397217, 2.5188722610473633, 2.512610912322998, 2.5554099082946777, 2.4574851989746094, 2.4087142944335938, 2.429823875427246, 2.4081263542175293, 2.5474934577941895, 2.38215970993042, 2.451215982437134, 2.5639870166778564, 2.3927762508392334, 2.603609085083008, 2.484323024749756, 2.39697265625, 2.4256083965301514]}, {"epoch_end": 0, "step": 391, "test_acc": 0.3403, "test_loss": 1.8708601318359375}, {"step": 500, "epoch": 1, "train_loss": 1.7038077116012573, "cos_r_negg": [0.5493754744529724, 0.01872267574071884, -0.05274730920791626, -0.0251530222594738, -0.02928505465388298, 0.021766304969787598, 0.010957082733511925, 0.02462802641093731, 0.007750675082206726, -0.03250172734260559, 0.013960735872387886, -0.03082090988755226, 0.021097920835018158, -0.027627158910036087, 0.027389727532863617, -0.005006357096135616, -0.04834308102726936, 0.01885084994137287, 0.019775716587901115, 0.0948190689086914, -0.061961255967617035, -0.017708592116832733, 0.01920778676867485, 0.05011627450585365, 0.12204429507255554, 0.04146486148238182, -0.030442122370004654, -0.0987292006611824, 0.040666453540325165, -0.04606577008962631, 0.041954271495342255, 0.07314084470272064, -0.0747339129447937, 0.029521167278289795, -0.04926997423171997, 0.02535322494804859, 0.025510795414447784, -0.0020328641403466463, -0.011173093691468239, -0.049767039716243744, -0.026263412088155746, 0.036890625953674316, -0.009711247868835926, -0.009342022240161896, 0.03803921863436699, 0.018719905987381935, 0.021647673100233078, 0.07000508904457092, -0.03713260591030121, -0.023778796195983887, -0.04196075350046158, -0.047701068222522736, -0.04460874944925308, 0.02074189856648445, 0.07091013342142105, -0.011496491730213165, 0.011327438987791538, 0.027212753891944885, -0.09790512174367905, -0.05965329706668854], "cos_apical_negg": [0.5493754744529724, 0.01872267574071884, -0.05274730920791626, -0.0251530222594738, -0.02928505465388298, 0.021766304969787598, 0.010957082733511925, 0.02462802641093731, 0.007750675082206726, -0.03250172734260559, 0.013960735872387886, -0.03082090988755226, 0.021097920835018158, -0.027627158910036087, 0.027389727532863617, -0.005006357096135616, -0.04834308102726936, 0.01885084994137287, 0.019775716587901115, 0.0948190689086914, -0.061961255967617035, -0.017708592116832733, 0.01920778676867485, 0.05011627450585365, 0.12204429507255554, 0.04146486148238182, -0.030442122370004654, -0.0987292006611824, 0.040666453540325165, -0.04606577008962631, 0.041954271495342255, 0.07314084470272064, -0.0747339129447937, 0.029521167278289795, -0.04926997423171997, 0.02535322494804859, 0.025510795414447784, -0.0020328641403466463, -0.011173093691468239, -0.049767039716243744, -0.026263412088155746, 0.036890625953674316, -0.009711247868835926, -0.009342022240161896, 0.03803921863436699, 0.018719905987381935, 0.021647673100233078, 0.07000508904457092, -0.03713260591030121, -0.023778796195983887, -0.04196075350046158, -0.047701068222522736, -0.04460874944925308, 0.02074189856648445, 0.07091013342142105, -0.011496491730213165, 0.011327438987791538, 0.027212753891944885, -0.09790512174367905, -0.05965329706668854], "cos_Ac_negg": [0.5493754744529724, 0.01872267574071884, -0.05274730920791626, -0.0251530222594738, -0.02928505465388298, 0.021766304969787598, 0.010957082733511925, 0.02462802641093731, 0.007750675082206726, -0.03250172734260559, 0.013960735872387886, -0.03082090988755226, 0.021097920835018158, -0.027627158910036087, 0.027389727532863617, -0.005006357096135616, -0.04834308102726936, 0.01885084994137287, 0.019775716587901115, 0.0948190689086914, -0.061961255967617035, -0.017708592116832733, 0.01920778676867485, 0.05011627450585365, 0.12204429507255554, 0.04146486148238182, -0.030442122370004654, -0.0987292006611824, 0.040666453540325165, -0.04606577008962631, 0.041954271495342255, 0.07314084470272064, -0.0747339129447937, 0.029521167278289795, -0.04926997423171997, 0.02535322494804859, 0.025510795414447784, -0.0020328641403466463, -0.011173093691468239, -0.049767039716243744, -0.026263412088155746, 0.036890625953674316, -0.009711247868835926, -0.009342022240161896, 0.03803921863436699, 0.018719905987381935, 0.021647673100233078, 0.07000508904457092, -0.03713260591030121, -0.023778796195983887, -0.04196075350046158, -0.047701068222522736, -0.04460874944925308, 0.02074189856648445, 0.07091013342142105, -0.011496491730213165, 0.011327438987791538, 0.027212753891944885, -0.09790512174367905, -0.05965329706668854], "r_norm": [2.1301116943359375, 2.232585906982422, 2.188831090927124, 2.1164536476135254, 1.99443781375885, 2.120631694793701, 2.0384206771850586, 2.1060214042663574, 2.2730677127838135, 2.1715474128723145, 2.1050007343292236, 2.220019578933716, 2.1526613235473633, 2.125145673751831, 2.2568631172180176, 2.1785054206848145, 2.227285385131836, 2.1672420501708984, 2.153294563293457, 2.2008848190307617, 2.291428327560425, 2.1129469871520996, 2.1159205436706543, 2.175632953643799, 2.148427963256836, 2.105003833770752, 2.2936763763427734, 2.1252517700195312, 2.0705881118774414, 2.286167621612549, 2.0095531940460205, 2.133378028869629, 2.109151601791382, 2.1417274475097656, 2.236701011657715, 2.1987481117248535, 2.0703554153442383, 2.081859588623047, 2.213632583618164, 2.1323516368865967, 2.211033344268799, 2.11452317237854, 2.1567158699035645, 2.117455005645752, 2.162360668182373, 2.1912946701049805, 2.211956024169922, 2.150634288787842, 2.105151891708374, 2.104519844055176, 2.1026124954223633, 2.2414767742156982, 2.0929369926452637, 2.1091196537017822, 2.2163383960723877, 2.0973386764526367, 2.248466730117798, 2.1563684940338135, 2.1060895919799805, 2.109330177307129]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3585, "test_loss": 1.8441361083984376}, {"step": 1000, "epoch": 2, "train_loss": 1.857997179031372, "cos_r_negg": [0.5343841314315796, 0.023018933832645416, -0.08571218699216843, -0.017818020656704903, -0.013029110617935658, 0.0748366266489029, 0.009163716807961464, 0.018014952540397644, 0.03471120819449425, 0.062045302242040634, 0.024813279509544373, -0.018575729802250862, 0.04900572821497917, -0.09401875734329224, 0.037270475178956985, -0.04430871456861496, -0.10339285433292389, 0.007345324382185936, -0.04356004297733307, 0.06882297992706299, -0.05566791445016861, 0.03943360969424248, 0.01875373348593712, 0.08718167245388031, 0.04017236828804016, -0.008893324993550777, 0.0021896613761782646, -0.005950205959379673, -0.009089072234928608, -0.007385949604213238, 0.08017701655626297, 0.027578268200159073, -0.06880444288253784, -0.007292254827916622, -0.015137006528675556, 0.049430668354034424, 0.059146419167518616, 0.018993591889739037, -0.0705374926328659, -0.02901827171444893, 0.0656881332397461, 0.014640504494309425, -0.024641098454594612, 0.03142322599887848, 0.02560984343290329, 0.018047764897346497, 0.022018013522028923, 0.019986581057310104, -0.0501948744058609, -0.03323870897293091, -0.06274554878473282, -0.041777465492486954, -0.04333692044019699, 0.02924686297774315, 0.04845748096704483, 0.02712920866906643, -0.02585148625075817, 0.03291165083646774, -0.044675443321466446, -0.06536038219928741], "cos_apical_negg": [0.5343841314315796, 0.023018933832645416, -0.08571218699216843, -0.017818020656704903, -0.013029110617935658, 0.0748366266489029, 0.009163716807961464, 0.018014952540397644, 0.03471120819449425, 0.062045302242040634, 0.024813279509544373, -0.018575729802250862, 0.04900572821497917, -0.09401875734329224, 0.037270475178956985, -0.04430871456861496, -0.10339285433292389, 0.007345324382185936, -0.04356004297733307, 0.06882297992706299, -0.05566791445016861, 0.03943360969424248, 0.01875373348593712, 0.08718167245388031, 0.04017236828804016, -0.008893324993550777, 0.0021896613761782646, -0.005950205959379673, -0.009089072234928608, -0.007385949604213238, 0.08017701655626297, 0.027578268200159073, -0.06880444288253784, -0.007292254827916622, -0.015137006528675556, 0.049430668354034424, 0.059146419167518616, 0.018993591889739037, -0.0705374926328659, -0.02901827171444893, 0.0656881332397461, 0.014640504494309425, -0.024641098454594612, 0.03142322599887848, 0.02560984343290329, 0.018047764897346497, 0.022018013522028923, 0.019986581057310104, -0.0501948744058609, -0.03323870897293091, -0.06274554878473282, -0.041777465492486954, -0.04333692044019699, 0.02924686297774315, 0.04845748096704483, 0.02712920866906643, -0.02585148625075817, 0.03291165083646774, -0.044675443321466446, -0.06536038219928741], "cos_Ac_negg": [0.5343841314315796, 0.023018933832645416, -0.08571218699216843, -0.017818020656704903, -0.013029110617935658, 0.0748366266489029, 0.009163716807961464, 0.018014952540397644, 0.03471120819449425, 0.062045302242040634, 0.024813279509544373, -0.018575729802250862, 0.04900572821497917, -0.09401875734329224, 0.037270475178956985, -0.04430871456861496, -0.10339285433292389, 0.007345324382185936, -0.04356004297733307, 0.06882297992706299, -0.05566791445016861, 0.03943360969424248, 0.01875373348593712, 0.08718167245388031, 0.04017236828804016, -0.008893324993550777, 0.0021896613761782646, -0.005950205959379673, -0.009089072234928608, -0.007385949604213238, 0.08017701655626297, 0.027578268200159073, -0.06880444288253784, -0.007292254827916622, -0.015137006528675556, 0.049430668354034424, 0.059146419167518616, 0.018993591889739037, -0.0705374926328659, -0.02901827171444893, 0.0656881332397461, 0.014640504494309425, -0.024641098454594612, 0.03142322599887848, 0.02560984343290329, 0.018047764897346497, 0.022018013522028923, 0.019986581057310104, -0.0501948744058609, -0.03323870897293091, -0.06274554878473282, -0.041777465492486954, -0.04333692044019699, 0.02924686297774315, 0.04845748096704483, 0.02712920866906643, -0.02585148625075817, 0.03291165083646774, -0.044675443321466446, -0.06536038219928741], "r_norm": [2.1409201622009277, 2.250765800476074, 2.2026870250701904, 2.1370391845703125, 2.0008046627044678, 2.1356072425842285, 2.059896945953369, 2.1249306201934814, 2.3059473037719727, 2.190365791320801, 2.12729811668396, 2.236894130706787, 2.168992042541504, 2.1360654830932617, 2.273104667663574, 2.197073459625244, 2.2362804412841797, 2.18412184715271, 2.1714978218078613, 2.222852945327759, 2.306452512741089, 2.132624387741089, 2.1374709606170654, 2.1788957118988037, 2.1593823432922363, 2.1206912994384766, 2.309680700302124, 2.147827625274658, 2.0874738693237305, 2.3094091415405273, 2.0352487564086914, 2.154064178466797, 2.1215386390686035, 2.1628928184509277, 2.253443956375122, 2.2141261100769043, 2.0794405937194824, 2.100123405456543, 2.235599994659424, 2.1525380611419678, 2.226761817932129, 2.1172890663146973, 2.1764984130859375, 2.138383626937866, 2.1824545860290527, 2.203514575958252, 2.224557399749756, 2.1673953533172607, 2.114952564239502, 2.128477096557617, 2.1164591312408447, 2.2617340087890625, 2.1204280853271484, 2.123225212097168, 2.2259621620178223, 2.117644786834717, 2.257298231124878, 2.171720504760742, 2.1138129234313965, 2.1265673637390137]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.346, "test_loss": 1.8643386474609376}, {"step": 1500, "epoch": 3, "train_loss": 1.8078371286392212, "cos_r_negg": [0.5336570739746094, 0.05931701511144638, -0.004795122426003218, -0.04091104492545128, -0.004944495391100645, 0.0733313038945198, 0.028732454404234886, -0.028997357934713364, 0.019572827965021133, -0.006624255329370499, 0.04258586838841438, -0.03980748727917671, 0.02592821791768074, -0.06463976949453354, 0.01925666071474552, -0.034768976271152496, -0.04177805036306381, -0.02030637487769127, -0.005816735327243805, 0.1192202940583229, -0.006012198515236378, 0.04902844876050949, 0.047489654272794724, 0.10890227556228638, 0.1260370910167694, 0.032090477645397186, 0.02169676497578621, -0.04004015773534775, 0.0039202249608933926, -0.024180863052606583, 0.027383621782064438, 0.061523422598838806, -0.08911024034023285, -0.03881463408470154, -0.004562738351523876, 0.06938697397708893, 0.045063890516757965, 0.024196378886699677, -0.024422064423561096, -0.01611250638961792, 0.07983332872390747, 0.03923346474766731, -0.016006378456950188, 0.01824045181274414, 0.0027705919928848743, -0.0370522066950798, 0.021477311849594116, 0.04933709278702736, -0.023096324875950813, -0.08095914870500565, -0.026619909331202507, -0.037193916738033295, -0.04080085828900337, 0.05228857323527336, 0.07540442049503326, 0.046373844146728516, 0.02178887277841568, 0.02086247317492962, -0.001101955771446228, -0.020234135910868645], "cos_apical_negg": [0.5336570739746094, 0.05931701511144638, -0.004795122426003218, -0.04091104492545128, -0.004944495391100645, 0.0733313038945198, 0.028732454404234886, -0.028997357934713364, 0.019572827965021133, -0.006624255329370499, 0.04258586838841438, -0.03980748727917671, 0.02592821791768074, -0.06463976949453354, 0.01925666071474552, -0.034768976271152496, -0.04177805036306381, -0.02030637487769127, -0.005816735327243805, 0.1192202940583229, -0.006012198515236378, 0.04902844876050949, 0.047489654272794724, 0.10890227556228638, 0.1260370910167694, 0.032090477645397186, 0.02169676497578621, -0.04004015773534775, 0.0039202249608933926, -0.024180863052606583, 0.027383621782064438, 0.061523422598838806, -0.08911024034023285, -0.03881463408470154, -0.004562738351523876, 0.06938697397708893, 0.045063890516757965, 0.024196378886699677, -0.024422064423561096, -0.01611250638961792, 0.07983332872390747, 0.03923346474766731, -0.016006378456950188, 0.01824045181274414, 0.0027705919928848743, -0.0370522066950798, 0.021477311849594116, 0.04933709278702736, -0.023096324875950813, -0.08095914870500565, -0.026619909331202507, -0.037193916738033295, -0.04080085828900337, 0.05228857323527336, 0.07540442049503326, 0.046373844146728516, 0.02178887277841568, 0.02086247317492962, -0.001101955771446228, -0.020234135910868645], "cos_Ac_negg": [0.5336570739746094, 0.05931701511144638, -0.004795122426003218, -0.04091104492545128, -0.004944495391100645, 0.0733313038945198, 0.028732454404234886, -0.028997357934713364, 0.019572827965021133, -0.006624255329370499, 0.04258586838841438, -0.03980748727917671, 0.02592821791768074, -0.06463976949453354, 0.01925666071474552, -0.034768976271152496, -0.04177805036306381, -0.02030637487769127, -0.005816735327243805, 0.1192202940583229, -0.006012198515236378, 0.04902844876050949, 0.047489654272794724, 0.10890227556228638, 0.1260370910167694, 0.032090477645397186, 0.02169676497578621, -0.04004015773534775, 0.0039202249608933926, -0.024180863052606583, 0.027383621782064438, 0.061523422598838806, -0.08911024034023285, -0.03881463408470154, -0.004562738351523876, 0.06938697397708893, 0.045063890516757965, 0.024196378886699677, -0.024422064423561096, -0.01611250638961792, 0.07983332872390747, 0.03923346474766731, -0.016006378456950188, 0.01824045181274414, 0.0027705919928848743, -0.0370522066950798, 0.021477311849594116, 0.04933709278702736, -0.023096324875950813, -0.08095914870500565, -0.026619909331202507, -0.037193916738033295, -0.04080085828900337, 0.05228857323527336, 0.07540442049503326, 0.046373844146728516, 0.02178887277841568, 0.02086247317492962, -0.001101955771446228, -0.020234135910868645], "r_norm": [2.160775661468506, 2.2687463760375977, 2.221665382385254, 2.153689384460449, 2.0242919921875, 2.1568546295166016, 2.0728518962860107, 2.1425580978393555, 2.307276964187622, 2.2065024375915527, 2.14194393157959, 2.2507224082946777, 2.176478385925293, 2.160116195678711, 2.2926833629608154, 2.2115936279296875, 2.2673709392547607, 2.200784206390381, 2.185403823852539, 2.24324893951416, 2.3234622478485107, 2.1513333320617676, 2.1534721851348877, 2.2044684886932373, 2.1821813583374023, 2.134456157684326, 2.3281219005584717, 2.1578588485717773, 2.1006298065185547, 2.3158438205718994, 2.0407919883728027, 2.1536338329315186, 2.1423592567443848, 2.179675817489624, 2.270850658416748, 2.22934627532959, 2.09849214553833, 2.1187362670898438, 2.2497034072875977, 2.1682066917419434, 2.2457528114318848, 2.1450448036193848, 2.1872382164001465, 2.1476597785949707, 2.1952481269836426, 2.226612091064453, 2.24898624420166, 2.1792969703674316, 2.1390697956085205, 2.144273281097412, 2.1311168670654297, 2.27933931350708, 2.126697063446045, 2.1475963592529297, 2.245152711868286, 2.1273446083068848, 2.280330181121826, 2.184453010559082, 2.1324613094329834, 2.138012170791626]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.3627, "test_loss": 1.8448671020507812}, {"epoch_end": 4, "step": 1955, "test_acc": 0.3731, "test_loss": 1.8088125122070313}], "final": {"test_acc": 0.3731, "test_loss": 1.8088125122070313, "wall_s": 41.05160164833069, "cos_r_negg": [0.521770715713501, 0.008525252342224121, -0.00947762094438076, 0.04014284908771515, 0.03533230721950531, 0.007415211759507656, 0.0013272948563098907, 0.060824908316135406, 0.021514687687158585, 0.023938609287142754, 0.07601390779018402, -0.061907485127449036, 0.04844936728477478, -0.023666687309741974, 0.040054649114608765, -0.03930802643299103, -0.021309133619070053, 0.01734125055372715, 0.015159038826823235, 0.06305719912052155, -0.05200711637735367, 0.03500454127788544, 0.031798359006643295, 0.042145244777202606, 0.08821828663349152, 0.04043489694595337, 0.04382440075278282, -0.004821183625608683, 0.013737156987190247, 0.0037198830395936966, 0.07937326282262802, 0.04212981089949608, -0.0645582526922226, 0.01643211394548416, 0.005947161465883255, 0.06351327896118164, 0.026483042165637016, 0.011229969561100006, -0.050125785171985626, 0.013528898358345032, 0.04318409413099289, 0.06815628707408905, -0.04041041433811188, 0.04399698227643967, -0.01782991737127304, -0.03676197677850723, 0.03483232110738754, 0.06307149678468704, -0.014615458436310291, -0.0490817129611969, -0.07918895781040192, -0.02049669437110424, -0.05999218672513962, 0.06805650144815445, 0.07911649346351624, 0.03073415346443653, 0.05748293176293373, 0.07361705601215363, 0.01881752908229828, 0.0017498719971626997], "cos_apical_negg": [0.521770715713501, 0.008525252342224121, -0.00947762094438076, 0.04014284908771515, 0.03533230721950531, 0.007415211759507656, 0.0013272948563098907, 0.060824908316135406, 0.021514687687158585, 0.023938609287142754, 0.07601390779018402, -0.061907485127449036, 0.04844936728477478, -0.023666687309741974, 0.040054649114608765, -0.03930802643299103, -0.021309133619070053, 0.01734125055372715, 0.015159038826823235, 0.06305719912052155, -0.05200711637735367, 0.03500454127788544, 0.031798359006643295, 0.042145244777202606, 0.08821828663349152, 0.04043489694595337, 0.04382440075278282, -0.004821183625608683, 0.013737156987190247, 0.0037198830395936966, 0.07937326282262802, 0.04212981089949608, -0.0645582526922226, 0.01643211394548416, 0.005947161465883255, 0.06351327896118164, 0.026483042165637016, 0.011229969561100006, -0.050125785171985626, 0.013528898358345032, 0.04318409413099289, 0.06815628707408905, -0.04041041433811188, 0.04399698227643967, -0.01782991737127304, -0.03676197677850723, 0.03483232110738754, 0.06307149678468704, -0.014615458436310291, -0.0490817129611969, -0.07918895781040192, -0.02049669437110424, -0.05999218672513962, 0.06805650144815445, 0.07911649346351624, 0.03073415346443653, 0.05748293176293373, 0.07361705601215363, 0.01881752908229828, 0.0017498719971626997], "cos_Ac_negg": [0.521770715713501, 0.008525252342224121, -0.00947762094438076, 0.04014284908771515, 0.03533230721950531, 0.007415211759507656, 0.0013272948563098907, 0.060824908316135406, 0.021514687687158585, 0.023938609287142754, 0.07601390779018402, -0.061907485127449036, 0.04844936728477478, -0.023666687309741974, 0.040054649114608765, -0.03930802643299103, -0.021309133619070053, 0.01734125055372715, 0.015159038826823235, 0.06305719912052155, -0.05200711637735367, 0.03500454127788544, 0.031798359006643295, 0.042145244777202606, 0.08821828663349152, 0.04043489694595337, 0.04382440075278282, -0.004821183625608683, 0.013737156987190247, 0.0037198830395936966, 0.07937326282262802, 0.04212981089949608, -0.0645582526922226, 0.01643211394548416, 0.005947161465883255, 0.06351327896118164, 0.026483042165637016, 0.011229969561100006, -0.050125785171985626, 0.013528898358345032, 0.04318409413099289, 0.06815628707408905, -0.04041041433811188, 0.04399698227643967, -0.01782991737127304, -0.03676197677850723, 0.03483232110738754, 0.06307149678468704, -0.014615458436310291, -0.0490817129611969, -0.07918895781040192, -0.02049669437110424, -0.05999218672513962, 0.06805650144815445, 0.07911649346351624, 0.03073415346443653, 0.05748293176293373, 0.07361705601215363, 0.01881752908229828, 0.0017498719971626997]}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_sdil_w64_d30_s0.json b/results/pilot_jac_cifar10_sdil_w64_d30_s0.json new file mode 100644 index 0000000..d3d8a55 --- /dev/null +++ b/results/pilot_jac_cifar10_sdil_w64_d30_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 30, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_sdil_w64_d30_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.4157285690307617, "cos_r_negg": [0.007795530371367931, -0.03972156345844269, 0.02516728639602661, 0.032594986259937286, -0.021284010261297226, -0.022270554676651955, 0.018770918250083923, -0.011221065185964108, 0.0958898589015007, -0.023709135130047798, -0.04670961946249008, 0.0031799222342669964, 0.00484514469280839, 0.023440927267074585, -0.017682580277323723, 0.03149745613336563, -0.01755579002201557, 0.004238690249621868, 0.003739067120477557, 0.03512764349579811, 0.04358594864606857, -0.009650608524680138, 0.03439425677061081, -0.03205307945609093, -0.05162773281335831, -0.04418618977069855, 0.019690409302711487, 0.007683158852159977, 0.05375661700963974, 0.044327251613140106], "cos_apical_negg": [0.007795530371367931, -0.03972156345844269, 0.02516728639602661, 0.032594986259937286, -0.021284010261297226, -0.022270554676651955, 0.018770918250083923, -0.011221065185964108, 0.0958898589015007, -0.023709135130047798, -0.04670961946249008, 0.0031799222342669964, 0.00484514469280839, 0.023440927267074585, -0.017682580277323723, 0.03149745613336563, -0.01755579002201557, 0.004238690249621868, 0.003739067120477557, 0.03512764349579811, 0.04358594864606857, -0.009650608524680138, 0.03439425677061081, -0.03205307945609093, -0.05162773281335831, -0.04418618977069855, 0.019690409302711487, 0.007683158852159977, 0.05375661700963974, 0.044327251613140106], "cos_Ac_negg": [0.007795530371367931, -0.03972156345844269, 0.02516728639602661, 0.032594986259937286, -0.021284010261297226, -0.022270554676651955, 0.018770918250083923, -0.011221065185964108, 0.0958898589015007, -0.023709135130047798, -0.04670961946249008, 0.0031799222342669964, 0.00484514469280839, 0.023440927267074585, -0.017682580277323723, 0.03149745613336563, -0.01755579002201557, 0.004238690249621868, 0.003739067120477557, 0.03512764349579811, 0.04358594864606857, -0.009650608524680138, 0.03439425677061081, -0.03205307945609093, -0.05162773281335831, -0.04418618977069855, 0.019690409302711487, 0.007683158852159977, 0.05375661700963974, 0.044327251613140106], "r_norm": [2.5859997272491455, 2.487873077392578, 2.358017683029175, 2.535337448120117, 2.3908329010009766, 2.4138553142547607, 2.5032689571380615, 2.4557106494903564, 2.3779029846191406, 2.3751931190490723, 2.451996326446533, 2.3805606365203857, 2.4780707359313965, 2.5016164779663086, 2.4434757232666016, 2.4289183616638184, 2.4294681549072266, 2.389345169067383, 2.382108449935913, 2.4187893867492676, 2.561915397644043, 2.449065685272217, 2.551858901977539, 2.4553465843200684, 2.4081687927246094, 2.4057531356811523, 2.3511159420013428, 2.398050308227539, 2.4193081855773926, 2.5638649463653564], "ldr": {"dL_sdil": 0.0014848709106445312, "dL_bp": -0.35697102546691895, "ratio": -0.004159639871898053}}, {"epoch_end": 0, "step": 391, "test_acc": 0.36, "test_loss": 1.8044174682617187}, {"step": 500, "epoch": 1, "train_loss": 1.7544350624084473, "cos_r_negg": [0.6653290390968323, 0.07077045738697052, 0.046043671667575836, 0.03851096332073212, 0.015489456243813038, 0.07029647380113602, 0.020347626879811287, 0.02587750181555748, 0.13888058066368103, 0.07940129190683365, 0.036375246942043304, 0.07947824895381927, 0.04108300805091858, 0.041909366846084595, 0.055439747869968414, 0.11433891952037811, 0.0001566256396472454, 0.04260788857936859, 0.08405902236700058, 0.0006076064892113209, 0.0595521479845047, 0.06639420986175537, 0.08550362288951874, 0.10366608947515488, 0.040724750608205795, 0.07333000004291534, 0.14777901768684387, 0.03469246253371239, 0.0794641375541687, 0.05990216135978699], "cos_apical_negg": [0.6653290390968323, 0.07077045738697052, 0.046043671667575836, 0.03851096332073212, 0.015489456243813038, 0.07029647380113602, 0.020347626879811287, 0.02587750181555748, 0.13888058066368103, 0.07940129190683365, 0.036375246942043304, 0.07947824895381927, 0.04108300805091858, 0.041909366846084595, 0.055439747869968414, 0.11433891952037811, 0.0001566256396472454, 0.04260788857936859, 0.08405902236700058, 0.0006076064892113209, 0.0595521479845047, 0.06639420986175537, 0.08550362288951874, 0.10366608947515488, 0.040724750608205795, 0.07333000004291534, 0.14777901768684387, 0.03469246253371239, 0.0794641375541687, 0.05990216135978699], "cos_Ac_negg": [0.6653290390968323, 0.07077045738697052, 0.046043671667575836, 0.03851096332073212, 0.015489456243813038, 0.07029647380113602, 0.020347626879811287, 0.02587750181555748, 0.13888058066368103, 0.07940129190683365, 0.036375246942043304, 0.07947824895381927, 0.04108300805091858, 0.041909366846084595, 0.055439747869968414, 0.11433891952037811, 0.0001566256396472454, 0.04260788857936859, 0.08405902236700058, 0.0006076064892113209, 0.0595521479845047, 0.06639420986175537, 0.08550362288951874, 0.10366608947515488, 0.040724750608205795, 0.07333000004291534, 0.14777901768684387, 0.03469246253371239, 0.0794641375541687, 0.05990216135978699], "r_norm": [1.8760464191436768, 1.7462239265441895, 1.6580451726913452, 1.7832438945770264, 1.6699025630950928, 1.6950693130493164, 1.7377784252166748, 1.7147564888000488, 1.6766102313995361, 1.673931360244751, 1.731095314025879, 1.6688055992126465, 1.753852367401123, 1.7483751773834229, 1.7181192636489868, 1.7302758693695068, 1.7059900760650635, 1.6811676025390625, 1.6618149280548096, 1.7175966501235962, 1.7833949327468872, 1.7352163791656494, 1.783334732055664, 1.7061405181884766, 1.7048929929733276, 1.6877436637878418, 1.6379642486572266, 1.7044821977615356, 1.701486349105835, 1.7973133325576782]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3931, "test_loss": 1.7496632690429688}, {"step": 1000, "epoch": 2, "train_loss": 1.7652791738510132, "cos_r_negg": [0.7455675005912781, 0.13867875933647156, 0.13353434205055237, 0.12310019135475159, 0.06930694729089737, 0.19699104130268097, 0.09868583083152771, 0.08995923399925232, 0.17063626646995544, 0.15953755378723145, 0.11872422695159912, 0.15358929336071014, 0.1210494190454483, 0.0893692672252655, 0.1626296043395996, 0.1403917521238327, 0.08300290256738663, 0.10640966147184372, 0.16039898991584778, 0.07700759172439575, 0.12020392715930939, 0.1384168118238449, 0.14931192994117737, 0.15331223607063293, 0.10677136480808258, 0.20391954481601715, 0.21600967645645142, 0.12095564603805542, 0.1786321997642517, 0.11213947832584381], "cos_apical_negg": [0.7455675005912781, 0.13867875933647156, 0.13353434205055237, 0.12310019135475159, 0.06930694729089737, 0.19699104130268097, 0.09868583083152771, 0.08995923399925232, 0.17063626646995544, 0.15953755378723145, 0.11872422695159912, 0.15358929336071014, 0.1210494190454483, 0.0893692672252655, 0.1626296043395996, 0.1403917521238327, 0.08300290256738663, 0.10640966147184372, 0.16039898991584778, 0.07700759172439575, 0.12020392715930939, 0.1384168118238449, 0.14931192994117737, 0.15331223607063293, 0.10677136480808258, 0.20391954481601715, 0.21600967645645142, 0.12095564603805542, 0.1786321997642517, 0.11213947832584381], "cos_Ac_negg": [0.7455675005912781, 0.13867875933647156, 0.13353434205055237, 0.12310019135475159, 0.06930694729089737, 0.19699104130268097, 0.09868583083152771, 0.08995923399925232, 0.17063626646995544, 0.15953755378723145, 0.11872422695159912, 0.15358929336071014, 0.1210494190454483, 0.0893692672252655, 0.1626296043395996, 0.1403917521238327, 0.08300290256738663, 0.10640966147184372, 0.16039898991584778, 0.07700759172439575, 0.12020392715930939, 0.1384168118238449, 0.14931192994117737, 0.15331223607063293, 0.10677136480808258, 0.20391954481601715, 0.21600967645645142, 0.12095564603805542, 0.1786321997642517, 0.11213947832584381], "r_norm": [1.5910654067993164, 1.4055867195129395, 1.3338172435760498, 1.4394001960754395, 1.3442482948303223, 1.3745750188827515, 1.3977023363113403, 1.3759074211120605, 1.3583508729934692, 1.356281042098999, 1.3894156217575073, 1.348914623260498, 1.4116623401641846, 1.4084699153900146, 1.3849666118621826, 1.3983466625213623, 1.3663556575775146, 1.3503215312957764, 1.3470991849899292, 1.3754619359970093, 1.4387788772583008, 1.396618127822876, 1.4419196844100952, 1.3856663703918457, 1.3714642524719238, 1.3673222064971924, 1.3337500095367432, 1.374590277671814, 1.373435378074646, 1.445796012878418]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.3952, "test_loss": 1.7153171020507811}, {"step": 1500, "epoch": 3, "train_loss": 1.5622105598449707, "cos_r_negg": [0.7900501489639282, 0.23095738887786865, 0.2347467541694641, 0.2343415766954422, 0.1593959629535675, 0.27783647179603577, 0.21201758086681366, 0.2417035698890686, 0.2442992627620697, 0.26645994186401367, 0.2338390350341797, 0.21624505519866943, 0.2291589379310608, 0.22390757501125336, 0.23851387202739716, 0.25256645679473877, 0.20062780380249023, 0.2368638515472412, 0.26529771089553833, 0.2094416320323944, 0.2076682448387146, 0.22922424972057343, 0.22596962749958038, 0.2822343707084656, 0.19691810011863708, 0.2956254184246063, 0.3513014018535614, 0.2733116149902344, 0.2675454318523407, 0.2186736762523651], "cos_apical_negg": [0.7900501489639282, 0.23095738887786865, 0.2347467541694641, 0.2343415766954422, 0.1593959629535675, 0.27783647179603577, 0.21201758086681366, 0.2417035698890686, 0.2442992627620697, 0.26645994186401367, 0.2338390350341797, 0.21624505519866943, 0.2291589379310608, 0.22390757501125336, 0.23851387202739716, 0.25256645679473877, 0.20062780380249023, 0.2368638515472412, 0.26529771089553833, 0.2094416320323944, 0.2076682448387146, 0.22922424972057343, 0.22596962749958038, 0.2822343707084656, 0.19691810011863708, 0.2956254184246063, 0.3513014018535614, 0.2733116149902344, 0.2675454318523407, 0.2186736762523651], "cos_Ac_negg": [0.7900501489639282, 0.23095738887786865, 0.2347467541694641, 0.2343415766954422, 0.1593959629535675, 0.27783647179603577, 0.21201758086681366, 0.2417035698890686, 0.2442992627620697, 0.26645994186401367, 0.2338390350341797, 0.21624505519866943, 0.2291589379310608, 0.22390757501125336, 0.23851387202739716, 0.25256645679473877, 0.20062780380249023, 0.2368638515472412, 0.26529771089553833, 0.2094416320323944, 0.2076682448387146, 0.22922424972057343, 0.22596962749958038, 0.2822343707084656, 0.19691810011863708, 0.2956254184246063, 0.3513014018535614, 0.2733116149902344, 0.2675454318523407, 0.2186736762523651], "r_norm": [1.3963406085968018, 1.1641191244125366, 1.1073403358459473, 1.1918562650680542, 1.1104576587677002, 1.1441190242767334, 1.1535261869430542, 1.130647897720337, 1.1298543214797974, 1.1300055980682373, 1.147794485092163, 1.1211740970611572, 1.1667977571487427, 1.1620769500732422, 1.151087760925293, 1.1575617790222168, 1.1221256256103516, 1.1212615966796875, 1.1218758821487427, 1.1313939094543457, 1.1885986328125, 1.1559741497039795, 1.200470209121704, 1.1557137966156006, 1.1320562362670898, 1.13889479637146, 1.1226190328598022, 1.1276708841323853, 1.1505897045135498, 1.1848069429397583]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4057, "test_loss": 1.7090366943359374}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4201, "test_loss": 1.6851649291992188}], "final": {"test_acc": 0.4201, "test_loss": 1.6851649291992188, "wall_s": 339.486483335495, "cos_r_negg": [0.7948374152183533, 0.3171462118625641, 0.32481592893600464, 0.31079748272895813, 0.29714739322662354, 0.37829869985580444, 0.287973552942276, 0.30488693714141846, 0.3360999524593353, 0.3544768691062927, 0.31089234352111816, 0.33214902877807617, 0.29504838585853577, 0.3175065815448761, 0.3597695231437683, 0.36076027154922485, 0.28100791573524475, 0.3558361530303955, 0.37889575958251953, 0.31262513995170593, 0.2987179756164551, 0.3624134659767151, 0.3272242546081543, 0.3705102205276489, 0.3145012855529785, 0.4177534282207489, 0.4370291829109192, 0.3744991421699524, 0.3817164897918701, 0.31987518072128296], "cos_apical_negg": [0.7948374152183533, 0.3171462118625641, 0.32481592893600464, 0.31079748272895813, 0.29714739322662354, 0.37829869985580444, 0.287973552942276, 0.30488693714141846, 0.3360999524593353, 0.3544768691062927, 0.31089234352111816, 0.33214902877807617, 0.29504838585853577, 0.3175065815448761, 0.3597695231437683, 0.36076027154922485, 0.28100791573524475, 0.3558361530303955, 0.37889575958251953, 0.31262513995170593, 0.2987179756164551, 0.3624134659767151, 0.3272242546081543, 0.3705102205276489, 0.3145012855529785, 0.4177534282207489, 0.4370291829109192, 0.3744991421699524, 0.3817164897918701, 0.31987518072128296], "cos_Ac_negg": [0.7948374152183533, 0.3171462118625641, 0.32481592893600464, 0.31079748272895813, 0.29714739322662354, 0.37829869985580444, 0.287973552942276, 0.30488693714141846, 0.3360999524593353, 0.3544768691062927, 0.31089234352111816, 0.33214902877807617, 0.29504838585853577, 0.3175065815448761, 0.3597695231437683, 0.36076027154922485, 0.28100791573524475, 0.3558361530303955, 0.37889575958251953, 0.31262513995170593, 0.2987179756164551, 0.3624134659767151, 0.3272242546081543, 0.3705102205276489, 0.3145012855529785, 0.4177534282207489, 0.4370291829109192, 0.3744991421699524, 0.3817164897918701, 0.31987518072128296]}}
\ No newline at end of file diff --git a/results/pilot_jac_cifar10_sdil_w64_d60_s0.json b/results/pilot_jac_cifar10_sdil_w64_d60_s0.json new file mode 100644 index 0000000..a6f018a --- /dev/null +++ b/results/pilot_jac_cifar10_sdil_w64_d60_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 60, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 8, "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_jac_cifar10_sdil_w64_d60_s0", "n_in": 3072}, "provenance": {"git_commit": "4856f33c47ae490d6688eef50db50a3d69766406", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.6154470443725586, "cos_r_negg": [0.008306834846735, 0.031316302716732025, -0.047569990158081055, -0.002081446349620819, -0.013024646788835526, -0.014964137226343155, -0.07074739038944244, -0.030616560950875282, -0.019917456433176994, -0.051495689898729324, 0.04967126250267029, -0.021022893488407135, 0.0029896735213696957, -0.05180514603853226, -0.05710602551698685, -0.06971040368080139, 0.08195842802524567, 0.04169908165931702, 0.052437759935855865, -0.03276614472270012, 0.023274565115571022, -0.008369792252779007, 0.07023535668849945, 0.018427327275276184, -0.008258171379566193, -0.011151166632771492, -0.0631258562207222, -0.004423070698976517, -0.06583043932914734, -0.03081006370484829, 0.043998681008815765, 0.003917967434972525, -0.023169618099927902, -0.0330444872379303, 0.003676089458167553, 0.020495446398854256, -0.006117891985923052, -9.540445171296597e-05, 0.030339309945702553, 0.016986183822155, 0.09264453500509262, 0.02000325545668602, -0.013069109059870243, 0.037566177546978, -0.04402650520205498, 0.00670284079387784, -0.04711200296878815, -0.0006688141729682684, -0.05795430392026901, 0.047791801393032074, 0.005219310522079468, -0.10099175572395325, 0.030650371685624123, 0.029977841302752495, -0.015634089708328247, 0.0419137142598629, 0.013834621757268906, -0.06069077551364899, -0.04862801730632782, -0.03141175955533981], "cos_apical_negg": [0.008306834846735, 0.031316302716732025, -0.047569990158081055, -0.002081446349620819, -0.013024646788835526, -0.014964137226343155, -0.07074739038944244, -0.030616560950875282, -0.019917456433176994, -0.051495689898729324, 0.04967126250267029, -0.021022893488407135, 0.0029896735213696957, -0.05180514603853226, -0.05710602551698685, -0.06971040368080139, 0.08195842802524567, 0.04169908165931702, 0.052437759935855865, -0.03276614472270012, 0.023274565115571022, -0.008369792252779007, 0.07023535668849945, 0.018427327275276184, -0.008258171379566193, -0.011151166632771492, -0.0631258562207222, -0.004423070698976517, -0.06583043932914734, -0.03081006370484829, 0.043998681008815765, 0.003917967434972525, -0.023169618099927902, -0.0330444872379303, 0.003676089458167553, 0.020495446398854256, -0.006117891985923052, -9.540445171296597e-05, 0.030339309945702553, 0.016986183822155, 0.09264453500509262, 0.02000325545668602, -0.013069109059870243, 0.037566177546978, -0.04402650520205498, 0.00670284079387784, -0.04711200296878815, -0.0006688141729682684, -0.05795430392026901, 0.047791801393032074, 0.005219310522079468, -0.10099175572395325, 0.030650371685624123, 0.029977841302752495, -0.015634089708328247, 0.0419137142598629, 0.013834621757268906, -0.06069077551364899, -0.04862801730632782, -0.03141175955533981], "cos_Ac_negg": [0.008306834846735, 0.031316302716732025, -0.047569990158081055, -0.002081446349620819, -0.013024646788835526, -0.014964137226343155, -0.07074739038944244, -0.030616560950875282, -0.019917456433176994, -0.051495689898729324, 0.04967126250267029, -0.021022893488407135, 0.0029896735213696957, -0.05180514603853226, -0.05710602551698685, -0.06971040368080139, 0.08195842802524567, 0.04169908165931702, 0.052437759935855865, -0.03276614472270012, 0.023274565115571022, -0.008369792252779007, 0.07023535668849945, 0.018427327275276184, -0.008258171379566193, -0.011151166632771492, -0.0631258562207222, -0.004423070698976517, -0.06583043932914734, -0.03081006370484829, 0.043998681008815765, 0.003917967434972525, -0.023169618099927902, -0.0330444872379303, 0.003676089458167553, 0.020495446398854256, -0.006117891985923052, -9.540445171296597e-05, 0.030339309945702553, 0.016986183822155, 0.09264453500509262, 0.02000325545668602, -0.013069109059870243, 0.037566177546978, -0.04402650520205498, 0.00670284079387784, -0.04711200296878815, -0.0006688141729682684, -0.05795430392026901, 0.047791801393032074, 0.005219310522079468, -0.10099175572395325, 0.030650371685624123, 0.029977841302752495, -0.015634089708328247, 0.0419137142598629, 0.013834621757268906, -0.06069077551364899, -0.04862801730632782, -0.03141175955533981], "r_norm": [2.475979804992676, 2.581467866897583, 2.5102410316467285, 2.431450843811035, 2.278761863708496, 2.466935157775879, 2.3775343894958496, 2.435208797454834, 2.5924670696258545, 2.5057992935180664, 2.40409255027771, 2.582052230834961, 2.4775142669677734, 2.4678969383239746, 2.603052854537964, 2.5098702907562256, 2.578503131866455, 2.519166946411133, 2.4873123168945312, 2.5275063514709473, 2.6223578453063965, 2.424740791320801, 2.4259703159332275, 2.507445812225342, 2.464304208755493, 2.429058074951172, 2.635713577270508, 2.448734998703003, 2.371245861053467, 2.6410045623779297, 2.341885566711426, 2.446690559387207, 2.4529337882995605, 2.473358392715454, 2.5734782218933105, 2.522921562194824, 2.4020309448242188, 2.3829708099365234, 2.540860414505005, 2.4576029777526855, 2.526998996734619, 2.4332480430603027, 2.478626251220703, 2.426945686340332, 2.5103676319122314, 2.5135018825531006, 2.553206443786621, 2.4561777114868164, 2.407553195953369, 2.429938554763794, 2.394683599472046, 2.547870635986328, 2.3780999183654785, 2.4377708435058594, 2.5554823875427246, 2.398036479949951, 2.608555316925049, 2.48563289642334, 2.399898052215576, 2.427046537399292], "ldr": {"dL_sdil": 0.014854192733764648, "dL_bp": -0.4264340400695801, "ratio": -0.03483350609473139}}, {"epoch_end": 0, "step": 391, "test_acc": 0.367, "test_loss": 1.80659228515625}, {"step": 500, "epoch": 1, "train_loss": 1.6646848917007446, "cos_r_negg": [0.6539371013641357, 0.04200909286737442, 0.028764337301254272, 0.04261940345168114, 0.07240334153175354, 0.08319608122110367, 0.07430893182754517, 0.08086370676755905, 0.05655743181705475, -0.004174408502876759, 0.08566032350063324, -0.02477475441992283, 0.030740534886717796, 0.05393870919942856, 0.06196482852101326, -0.013373151421546936, 0.02046332322061062, 0.05083515867590904, 0.07628284394741058, 0.10322341322898865, 0.013860978186130524, 0.06720785796642303, 0.11314959824085236, 0.08072827756404877, 0.11246846616268158, 0.05424723029136658, 0.0593956783413887, -0.010331064462661743, 0.054642125964164734, -0.0019361046142876148, 0.10925623774528503, 0.11035873740911484, -0.0018345494754612446, 0.039020128548145294, 0.039032064378261566, 0.10142432153224945, 0.05504220351576805, 0.09621832519769669, 0.010217140428721905, 0.0201725997030735, 0.07264318317174911, 0.09278057515621185, 0.03654636815190315, 0.07242821902036667, 0.0031978159677237272, 0.034640245139598846, 0.018171407282352448, 0.10636558383703232, 0.013465621508657932, 0.0339340940117836, -0.04428789019584656, 0.020617473870515823, 0.06526491045951843, 0.1225474551320076, 0.12936189770698547, 0.07360360026359558, 0.09782689809799194, 0.08423000574111938, -0.007182752713561058, 0.012765337713062763], "cos_apical_negg": [0.6539371013641357, 0.04200909286737442, 0.028764337301254272, 0.04261940345168114, 0.07240334153175354, 0.08319608122110367, 0.07430893182754517, 0.08086370676755905, 0.05655743181705475, -0.004174408502876759, 0.08566032350063324, -0.02477475441992283, 0.030740534886717796, 0.05393870919942856, 0.06196482852101326, -0.013373151421546936, 0.02046332322061062, 0.05083515867590904, 0.07628284394741058, 0.10322341322898865, 0.013860978186130524, 0.06720785796642303, 0.11314959824085236, 0.08072827756404877, 0.11246846616268158, 0.05424723029136658, 0.0593956783413887, -0.010331064462661743, 0.054642125964164734, -0.0019361046142876148, 0.10925623774528503, 0.11035873740911484, -0.0018345494754612446, 0.039020128548145294, 0.039032064378261566, 0.10142432153224945, 0.05504220351576805, 0.09621832519769669, 0.010217140428721905, 0.0201725997030735, 0.07264318317174911, 0.09278057515621185, 0.03654636815190315, 0.07242821902036667, 0.0031978159677237272, 0.034640245139598846, 0.018171407282352448, 0.10636558383703232, 0.013465621508657932, 0.0339340940117836, -0.04428789019584656, 0.020617473870515823, 0.06526491045951843, 0.1225474551320076, 0.12936189770698547, 0.07360360026359558, 0.09782689809799194, 0.08423000574111938, -0.007182752713561058, 0.012765337713062763], "cos_Ac_negg": [0.6539371013641357, 0.04200909286737442, 0.028764337301254272, 0.04261940345168114, 0.07240334153175354, 0.08319608122110367, 0.07430893182754517, 0.08086370676755905, 0.05655743181705475, -0.004174408502876759, 0.08566032350063324, -0.02477475441992283, 0.030740534886717796, 0.05393870919942856, 0.06196482852101326, -0.013373151421546936, 0.02046332322061062, 0.05083515867590904, 0.07628284394741058, 0.10322341322898865, 0.013860978186130524, 0.06720785796642303, 0.11314959824085236, 0.08072827756404877, 0.11246846616268158, 0.05424723029136658, 0.0593956783413887, -0.010331064462661743, 0.054642125964164734, -0.0019361046142876148, 0.10925623774528503, 0.11035873740911484, -0.0018345494754612446, 0.039020128548145294, 0.039032064378261566, 0.10142432153224945, 0.05504220351576805, 0.09621832519769669, 0.010217140428721905, 0.0201725997030735, 0.07264318317174911, 0.09278057515621185, 0.03654636815190315, 0.07242821902036667, 0.0031978159677237272, 0.034640245139598846, 0.018171407282352448, 0.10636558383703232, 0.013465621508657932, 0.0339340940117836, -0.04428789019584656, 0.020617473870515823, 0.06526491045951843, 0.1225474551320076, 0.12936189770698547, 0.07360360026359558, 0.09782689809799194, 0.08423000574111938, -0.007182752713561058, 0.012765337713062763], "r_norm": [1.7559453248977661, 1.7700048685073853, 1.7273797988891602, 1.6780471801757812, 1.5768959522247314, 1.6862986087799072, 1.6167422533035278, 1.6688613891601562, 1.7972103357315063, 1.7065937519073486, 1.678663969039917, 1.7558720111846924, 1.6995190382003784, 1.6855883598327637, 1.7889618873596191, 1.716568946838379, 1.764819860458374, 1.7146646976470947, 1.713696002960205, 1.7507054805755615, 1.8059900999069214, 1.673990249633789, 1.6869001388549805, 1.7217738628387451, 1.7060068845748901, 1.668450117111206, 1.818056344985962, 1.6814358234405518, 1.631054162979126, 1.804546594619751, 1.603561282157898, 1.69077730178833, 1.669722080230713, 1.6955254077911377, 1.7657537460327148, 1.7482422590255737, 1.6436355113983154, 1.6500483751296997, 1.7496362924575806, 1.6985191106796265, 1.7551921606063843, 1.6763815879821777, 1.7084565162658691, 1.6831642389297485, 1.7140097618103027, 1.7289087772369385, 1.744025468826294, 1.6992907524108887, 1.653989315032959, 1.6770350933074951, 1.6493605375289917, 1.7642641067504883, 1.6516772508621216, 1.6782639026641846, 1.761060118675232, 1.6609342098236084, 1.786983609199524, 1.7080414295196533, 1.6569240093231201, 1.6682915687561035]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3954, "test_loss": 1.7439866088867189}, {"step": 1000, "epoch": 2, "train_loss": 1.7360448837280273, "cos_r_negg": [0.7485468983650208, 0.15854232013225555, 0.062474995851516724, 0.08679990470409393, 0.13997483253479004, 0.10376215726137161, 0.13266262412071228, 0.10948528349399567, 0.12351656705141068, 0.07324019074440002, 0.11113777756690979, 0.036116279661655426, 0.149004727602005, 0.1030762642621994, 0.15905502438545227, 0.042355217039585114, 0.07570943236351013, 0.15692907571792603, 0.1737491488456726, 0.17710107564926147, 0.1156308576464653, 0.14633019268512726, 0.15874920785427094, 0.20415669679641724, 0.16529160737991333, 0.19328835606575012, 0.12919190526008606, 0.06598785519599915, 0.09985564649105072, 0.06557320058345795, 0.1835542917251587, 0.1722496747970581, 0.11013628542423248, 0.09385110437870026, 0.15136876702308655, 0.11882132291793823, 0.15022887289524078, 0.1809040755033493, 0.10653582960367203, 0.07725328207015991, 0.12757611274719238, 0.13994623720645905, 0.08367457985877991, 0.1468907594680786, 0.08826358616352081, 0.123940609395504, 0.10189810395240784, 0.15440917015075684, 0.1249677985906601, 0.11273837834596634, 0.02271694876253605, 0.1308736801147461, 0.0873950719833374, 0.18731771409511566, 0.18670541048049927, 0.17988982796669006, 0.14291013777256012, 0.1819171905517578, 0.08047103881835938, 0.0646289587020874], "cos_apical_negg": [0.7485468983650208, 0.15854232013225555, 0.062474995851516724, 0.08679990470409393, 0.13997483253479004, 0.10376215726137161, 0.13266262412071228, 0.10948528349399567, 0.12351656705141068, 0.07324019074440002, 0.11113777756690979, 0.036116279661655426, 0.149004727602005, 0.1030762642621994, 0.15905502438545227, 0.042355217039585114, 0.07570943236351013, 0.15692907571792603, 0.1737491488456726, 0.17710107564926147, 0.1156308576464653, 0.14633019268512726, 0.15874920785427094, 0.20415669679641724, 0.16529160737991333, 0.19328835606575012, 0.12919190526008606, 0.06598785519599915, 0.09985564649105072, 0.06557320058345795, 0.1835542917251587, 0.1722496747970581, 0.11013628542423248, 0.09385110437870026, 0.15136876702308655, 0.11882132291793823, 0.15022887289524078, 0.1809040755033493, 0.10653582960367203, 0.07725328207015991, 0.12757611274719238, 0.13994623720645905, 0.08367457985877991, 0.1468907594680786, 0.08826358616352081, 0.123940609395504, 0.10189810395240784, 0.15440917015075684, 0.1249677985906601, 0.11273837834596634, 0.02271694876253605, 0.1308736801147461, 0.0873950719833374, 0.18731771409511566, 0.18670541048049927, 0.17988982796669006, 0.14291013777256012, 0.1819171905517578, 0.08047103881835938, 0.0646289587020874], "cos_Ac_negg": [0.7485468983650208, 0.15854232013225555, 0.062474995851516724, 0.08679990470409393, 0.13997483253479004, 0.10376215726137161, 0.13266262412071228, 0.10948528349399567, 0.12351656705141068, 0.07324019074440002, 0.11113777756690979, 0.036116279661655426, 0.149004727602005, 0.1030762642621994, 0.15905502438545227, 0.042355217039585114, 0.07570943236351013, 0.15692907571792603, 0.1737491488456726, 0.17710107564926147, 0.1156308576464653, 0.14633019268512726, 0.15874920785427094, 0.20415669679641724, 0.16529160737991333, 0.19328835606575012, 0.12919190526008606, 0.06598785519599915, 0.09985564649105072, 0.06557320058345795, 0.1835542917251587, 0.1722496747970581, 0.11013628542423248, 0.09385110437870026, 0.15136876702308655, 0.11882132291793823, 0.15022887289524078, 0.1809040755033493, 0.10653582960367203, 0.07725328207015991, 0.12757611274719238, 0.13994623720645905, 0.08367457985877991, 0.1468907594680786, 0.08826358616352081, 0.123940609395504, 0.10189810395240784, 0.15440917015075684, 0.1249677985906601, 0.11273837834596634, 0.02271694876253605, 0.1308736801147461, 0.0873950719833374, 0.18731771409511566, 0.18670541048049927, 0.17988982796669006, 0.14291013777256012, 0.1819171905517578, 0.08047103881835938, 0.0646289587020874], "r_norm": [1.526496410369873, 1.454730749130249, 1.417557716369629, 1.3737268447875977, 1.3011536598205566, 1.388192892074585, 1.34401273727417, 1.37082839012146, 1.4874067306518555, 1.4026169776916504, 1.3795323371887207, 1.4410927295684814, 1.401087999343872, 1.3885788917541504, 1.4777989387512207, 1.4032313823699951, 1.4476779699325562, 1.418686866760254, 1.4130617380142212, 1.4416478872299194, 1.4804548025131226, 1.381870150566101, 1.394826889038086, 1.4226784706115723, 1.4114667177200317, 1.3773810863494873, 1.498025894165039, 1.3775441646575928, 1.3400335311889648, 1.4789161682128906, 1.3314495086669922, 1.3977910280227661, 1.3719788789749146, 1.3888976573944092, 1.4465715885162354, 1.442132830619812, 1.359398603439331, 1.3593394756317139, 1.4299870729446411, 1.3919024467468262, 1.4448397159576416, 1.3801032304763794, 1.40386962890625, 1.384777545928955, 1.4049603939056396, 1.417431354522705, 1.4311861991882324, 1.4060568809509277, 1.3540253639221191, 1.3776973485946655, 1.3376346826553345, 1.4411137104034424, 1.3559750318527222, 1.3854596614837646, 1.4516289234161377, 1.3701173067092896, 1.473273515701294, 1.4066729545593262, 1.3596673011779785, 1.3602051734924316]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.394, "test_loss": 1.7281866455078125}, {"step": 1500, "epoch": 3, "train_loss": 1.6272435188293457, "cos_r_negg": [0.7542884349822998, 0.2393415868282318, 0.18157801032066345, 0.1748751699924469, 0.2554659843444824, 0.1990557610988617, 0.18717333674430847, 0.18983498215675354, 0.23196369409561157, 0.1892457902431488, 0.20655326545238495, 0.11088135093450546, 0.20243357121944427, 0.2163824439048767, 0.24075117707252502, 0.12869198620319366, 0.1643744707107544, 0.2254430055618286, 0.24041400849819183, 0.2654344141483307, 0.18127042055130005, 0.23409093916416168, 0.2414664924144745, 0.2651432156562805, 0.2950062155723572, 0.3032069206237793, 0.20758852362632751, 0.1423550397157669, 0.24695803225040436, 0.1525401473045349, 0.2697913646697998, 0.23836590349674225, 0.16311274468898773, 0.17374326288700104, 0.2351670265197754, 0.24086111783981323, 0.24191555380821228, 0.2446436882019043, 0.17667661607265472, 0.186082124710083, 0.21416714787483215, 0.2234211564064026, 0.20322266221046448, 0.19883489608764648, 0.2246798723936081, 0.21545150876045227, 0.18748345971107483, 0.22947071492671967, 0.20121879875659943, 0.2156158685684204, 0.15268591046333313, 0.24747590720653534, 0.17483191192150116, 0.2910960614681244, 0.27153027057647705, 0.29465752840042114, 0.2701978385448456, 0.26268184185028076, 0.20438987016677856, 0.1329614818096161], "cos_apical_negg": [0.7542884349822998, 0.2393415868282318, 0.18157801032066345, 0.1748751699924469, 0.2554659843444824, 0.1990557610988617, 0.18717333674430847, 0.18983498215675354, 0.23196369409561157, 0.1892457902431488, 0.20655326545238495, 0.11088135093450546, 0.20243357121944427, 0.2163824439048767, 0.24075117707252502, 0.12869198620319366, 0.1643744707107544, 0.2254430055618286, 0.24041400849819183, 0.2654344141483307, 0.18127042055130005, 0.23409093916416168, 0.2414664924144745, 0.2651432156562805, 0.2950062155723572, 0.3032069206237793, 0.20758852362632751, 0.1423550397157669, 0.24695803225040436, 0.1525401473045349, 0.2697913646697998, 0.23836590349674225, 0.16311274468898773, 0.17374326288700104, 0.2351670265197754, 0.24086111783981323, 0.24191555380821228, 0.2446436882019043, 0.17667661607265472, 0.186082124710083, 0.21416714787483215, 0.2234211564064026, 0.20322266221046448, 0.19883489608764648, 0.2246798723936081, 0.21545150876045227, 0.18748345971107483, 0.22947071492671967, 0.20121879875659943, 0.2156158685684204, 0.15268591046333313, 0.24747590720653534, 0.17483191192150116, 0.2910960614681244, 0.27153027057647705, 0.29465752840042114, 0.2701978385448456, 0.26268184185028076, 0.20438987016677856, 0.1329614818096161], "cos_Ac_negg": [0.7542884349822998, 0.2393415868282318, 0.18157801032066345, 0.1748751699924469, 0.2554659843444824, 0.1990557610988617, 0.18717333674430847, 0.18983498215675354, 0.23196369409561157, 0.1892457902431488, 0.20655326545238495, 0.11088135093450546, 0.20243357121944427, 0.2163824439048767, 0.24075117707252502, 0.12869198620319366, 0.1643744707107544, 0.2254430055618286, 0.24041400849819183, 0.2654344141483307, 0.18127042055130005, 0.23409093916416168, 0.2414664924144745, 0.2651432156562805, 0.2950062155723572, 0.3032069206237793, 0.20758852362632751, 0.1423550397157669, 0.24695803225040436, 0.1525401473045349, 0.2697913646697998, 0.23836590349674225, 0.16311274468898773, 0.17374326288700104, 0.2351670265197754, 0.24086111783981323, 0.24191555380821228, 0.2446436882019043, 0.17667661607265472, 0.186082124710083, 0.21416714787483215, 0.2234211564064026, 0.20322266221046448, 0.19883489608764648, 0.2246798723936081, 0.21545150876045227, 0.18748345971107483, 0.22947071492671967, 0.20121879875659943, 0.2156158685684204, 0.15268591046333313, 0.24747590720653534, 0.17483191192150116, 0.2910960614681244, 0.27153027057647705, 0.29465752840042114, 0.2701978385448456, 0.26268184185028076, 0.20438987016677856, 0.1329614818096161], "r_norm": [1.3328654766082764, 1.1940767765045166, 1.1455750465393066, 1.1167540550231934, 1.068833589553833, 1.1384071111679077, 1.1001887321472168, 1.1196691989898682, 1.2072861194610596, 1.1445355415344238, 1.1336579322814941, 1.170964241027832, 1.1418182849884033, 1.1300058364868164, 1.211554765701294, 1.1296662092208862, 1.1769604682922363, 1.1637969017028809, 1.158700704574585, 1.182386875152588, 1.2025408744812012, 1.1274592876434326, 1.1469311714172363, 1.1747169494628906, 1.1591014862060547, 1.1338791847229004, 1.2290623188018799, 1.1148293018341064, 1.098484992980957, 1.1998419761657715, 1.1065282821655273, 1.1466820240020752, 1.118770718574524, 1.1355831623077393, 1.185559630393982, 1.1830594539642334, 1.1235426664352417, 1.1169803142547607, 1.161017894744873, 1.1422295570373535, 1.1802475452423096, 1.1340126991271973, 1.1426594257354736, 1.1292616128921509, 1.149385929107666, 1.1572332382202148, 1.1674866676330566, 1.143965244293213, 1.1051650047302246, 1.1141518354415894, 1.0870299339294434, 1.1786168813705444, 1.0947458744049072, 1.1474294662475586, 1.1961817741394043, 1.1247966289520264, 1.2138041257858276, 1.1578238010406494, 1.101365327835083, 1.101654052734375]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4111, "test_loss": 1.7163718872070313}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4079, "test_loss": 1.698351953125}], "final": {"test_acc": 0.4079, "test_loss": 1.698351953125, "wall_s": 1238.5222442150116, "cos_r_negg": [0.7825913429260254, 0.32893261313438416, 0.2500309944152832, 0.2823156714439392, 0.34893763065338135, 0.3250320255756378, 0.30813848972320557, 0.29275351762771606, 0.32152295112609863, 0.2688843607902527, 0.33976513147354126, 0.21493855118751526, 0.32880687713623047, 0.3047349154949188, 0.31467074155807495, 0.22618725895881653, 0.2532234191894531, 0.3205098509788513, 0.3595099449157715, 0.38603073358535767, 0.30192768573760986, 0.3239319920539856, 0.3560585677623749, 0.3739151358604431, 0.39094239473342896, 0.37151283025741577, 0.2948240041732788, 0.25084322690963745, 0.3263080418109894, 0.24757586419582367, 0.3797128200531006, 0.3600878119468689, 0.2767699658870697, 0.30135297775268555, 0.347915381193161, 0.32806649804115295, 0.3735722601413727, 0.36435800790786743, 0.32316476106643677, 0.30365413427352905, 0.3476790487766266, 0.31463634967803955, 0.2905779778957367, 0.3239128589630127, 0.30373090505599976, 0.3160437345504761, 0.31759393215179443, 0.33476683497428894, 0.33470410108566284, 0.3102900981903076, 0.27645355463027954, 0.3236931562423706, 0.27881601452827454, 0.3689236044883728, 0.36155179142951965, 0.3841766119003296, 0.3666810393333435, 0.39142778515815735, 0.28567004203796387, 0.27448493242263794], "cos_apical_negg": [0.7825913429260254, 0.32893261313438416, 0.2500309944152832, 0.2823156714439392, 0.34893763065338135, 0.3250320255756378, 0.30813848972320557, 0.29275351762771606, 0.32152295112609863, 0.2688843607902527, 0.33976513147354126, 0.21493855118751526, 0.32880687713623047, 0.3047349154949188, 0.31467074155807495, 0.22618725895881653, 0.2532234191894531, 0.3205098509788513, 0.3595099449157715, 0.38603073358535767, 0.30192768573760986, 0.3239319920539856, 0.3560585677623749, 0.3739151358604431, 0.39094239473342896, 0.37151283025741577, 0.2948240041732788, 0.25084322690963745, 0.3263080418109894, 0.24757586419582367, 0.3797128200531006, 0.3600878119468689, 0.2767699658870697, 0.30135297775268555, 0.347915381193161, 0.32806649804115295, 0.3735722601413727, 0.36435800790786743, 0.32316476106643677, 0.30365413427352905, 0.3476790487766266, 0.31463634967803955, 0.2905779778957367, 0.3239128589630127, 0.30373090505599976, 0.3160437345504761, 0.31759393215179443, 0.33476683497428894, 0.33470410108566284, 0.3102900981903076, 0.27645355463027954, 0.3236931562423706, 0.27881601452827454, 0.3689236044883728, 0.36155179142951965, 0.3841766119003296, 0.3666810393333435, 0.39142778515815735, 0.28567004203796387, 0.27448493242263794], "cos_Ac_negg": [0.7825913429260254, 0.32893261313438416, 0.2500309944152832, 0.2823156714439392, 0.34893763065338135, 0.3250320255756378, 0.30813848972320557, 0.29275351762771606, 0.32152295112609863, 0.2688843607902527, 0.33976513147354126, 0.21493855118751526, 0.32880687713623047, 0.3047349154949188, 0.31467074155807495, 0.22618725895881653, 0.2532234191894531, 0.3205098509788513, 0.3595099449157715, 0.38603073358535767, 0.30192768573760986, 0.3239319920539856, 0.3560585677623749, 0.3739151358604431, 0.39094239473342896, 0.37151283025741577, 0.2948240041732788, 0.25084322690963745, 0.3263080418109894, 0.24757586419582367, 0.3797128200531006, 0.3600878119468689, 0.2767699658870697, 0.30135297775268555, 0.347915381193161, 0.32806649804115295, 0.3735722601413727, 0.36435800790786743, 0.32316476106643677, 0.30365413427352905, 0.3476790487766266, 0.31463634967803955, 0.2905779778957367, 0.3239128589630127, 0.30373090505599976, 0.3160437345504761, 0.31759393215179443, 0.33476683497428894, 0.33470410108566284, 0.3102900981903076, 0.27645355463027954, 0.3236931562423706, 0.27881601452827454, 0.3689236044883728, 0.36155179142951965, 0.3841766119003296, 0.3666810393333435, 0.39142778515815735, 0.28567004203796387, 0.27448493242263794]}}
\ No newline at end of file diff --git a/results/pilot_sim_cifar10_sdil_w64_d30_s0.json b/results/pilot_sim_cifar10_sdil_w64_d30_s0.json new file mode 100644 index 0000000..c58742b --- /dev/null +++ b/results/pilot_sim_cifar10_sdil_w64_d30_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 30, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 16, "pert_mode": "simultaneous", "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_sim_cifar10_sdil_w64_d30_s0", "n_in": 3072}, "provenance": {"git_commit": "0f107d1ee4e750421a0e95f8bd06bf0f629d2fa7", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.4157285690307617, "cos_r_negg": [0.007889339700341225, -0.03945823386311531, 0.025249317288398743, 0.032812464982271194, -0.021251637488603592, -0.022359423339366913, 0.01877300813794136, -0.01140593271702528, 0.0959528237581253, -0.02352067083120346, -0.046616435050964355, 0.0034567127004265785, 0.004958100151270628, 0.023545071482658386, -0.01747933030128479, 0.03165106475353241, -0.017468497157096863, 0.004206513054668903, 0.003664806019514799, 0.03509009629487991, 0.04356950521469116, -0.009784087538719177, 0.03444090113043785, -0.03212546557188034, -0.05166994035243988, -0.04420970752835274, 0.019850967451930046, 0.00775631470605731, 0.0540740005671978, 0.04434435814619064], "cos_apical_negg": [0.007889339700341225, -0.03945823386311531, 0.025249317288398743, 0.032812464982271194, -0.021251637488603592, -0.022359423339366913, 0.01877300813794136, -0.01140593271702528, 0.0959528237581253, -0.02352067083120346, -0.046616435050964355, 0.0034567127004265785, 0.004958100151270628, 0.023545071482658386, -0.01747933030128479, 0.03165106475353241, -0.017468497157096863, 0.004206513054668903, 0.003664806019514799, 0.03509009629487991, 0.04356950521469116, -0.009784087538719177, 0.03444090113043785, -0.03212546557188034, -0.05166994035243988, -0.04420970752835274, 0.019850967451930046, 0.00775631470605731, 0.0540740005671978, 0.04434435814619064], "cos_Ac_negg": [0.007889339700341225, -0.03945823386311531, 0.025249317288398743, 0.032812464982271194, -0.021251637488603592, -0.022359423339366913, 0.01877300813794136, -0.01140593271702528, 0.0959528237581253, -0.02352067083120346, -0.046616435050964355, 0.0034567127004265785, 0.004958100151270628, 0.023545071482658386, -0.01747933030128479, 0.03165106475353241, -0.017468497157096863, 0.004206513054668903, 0.003664806019514799, 0.03509009629487991, 0.04356950521469116, -0.009784087538719177, 0.03444090113043785, -0.03212546557188034, -0.05166994035243988, -0.04420970752835274, 0.019850967451930046, 0.00775631470605731, 0.0540740005671978, 0.04434435814619064], "r_norm": [2.5863919258117676, 2.487764835357666, 2.358147621154785, 2.5351524353027344, 2.390735149383545, 2.414008855819702, 2.5036239624023438, 2.4557342529296875, 2.378182888031006, 2.3751134872436523, 2.45259952545166, 2.380277156829834, 2.4780802726745605, 2.5011937618255615, 2.443186044692993, 2.4289400577545166, 2.429619550704956, 2.3888304233551025, 2.382321357727051, 2.4184184074401855, 2.562037467956543, 2.449069023132324, 2.5521750450134277, 2.4552977085113525, 2.408527135848999, 2.4052858352661133, 2.351199150085449, 2.398348808288574, 2.4193458557128906, 2.563992500305176], "ldr": {"dL_sdil": 0.0013036727905273438, "dL_bp": -0.35697102546691895, "ratio": -0.0036520409151474874}}, {"epoch_end": 0, "step": 391, "test_acc": 0.367, "test_loss": 1.8024296630859376}, {"step": 500, "epoch": 1, "train_loss": 1.7450627088546753, "cos_r_negg": [0.6449319124221802, 0.05046190321445465, 0.01704675517976284, 0.11872923374176025, 0.07285906374454498, 0.11415785551071167, 0.060138579457998276, 0.025059571489691734, 0.10475346446037292, 0.08032901585102081, 0.02657567709684372, 0.044159214943647385, 0.01127505674958229, 0.040587302297353745, 0.04210765287280083, 0.0895003080368042, 0.018399164080619812, 0.037577006965875626, 0.022510327398777008, 0.014514397829771042, 0.03587838262319565, 0.05064118653535843, 0.10795119404792786, 0.06375892460346222, 0.013818235136568546, 0.0673866868019104, 0.10845224559307098, 0.048314422369003296, 0.054259222000837326, 0.07402704656124115], "cos_apical_negg": [0.6449319124221802, 0.05046190321445465, 0.01704675517976284, 0.11872923374176025, 0.07285906374454498, 0.11415785551071167, 0.060138579457998276, 0.025059571489691734, 0.10475346446037292, 0.08032901585102081, 0.02657567709684372, 0.044159214943647385, 0.01127505674958229, 0.040587302297353745, 0.04210765287280083, 0.0895003080368042, 0.018399164080619812, 0.037577006965875626, 0.022510327398777008, 0.014514397829771042, 0.03587838262319565, 0.05064118653535843, 0.10795119404792786, 0.06375892460346222, 0.013818235136568546, 0.0673866868019104, 0.10845224559307098, 0.048314422369003296, 0.054259222000837326, 0.07402704656124115], "cos_Ac_negg": [0.6449319124221802, 0.05046190321445465, 0.01704675517976284, 0.11872923374176025, 0.07285906374454498, 0.11415785551071167, 0.060138579457998276, 0.025059571489691734, 0.10475346446037292, 0.08032901585102081, 0.02657567709684372, 0.044159214943647385, 0.01127505674958229, 0.040587302297353745, 0.04210765287280083, 0.0895003080368042, 0.018399164080619812, 0.037577006965875626, 0.022510327398777008, 0.014514397829771042, 0.03587838262319565, 0.05064118653535843, 0.10795119404792786, 0.06375892460346222, 0.013818235136568546, 0.0673866868019104, 0.10845224559307098, 0.048314422369003296, 0.054259222000837326, 0.07402704656124115], "r_norm": [1.8447617292404175, 1.716846227645874, 1.6359447240829468, 1.7626714706420898, 1.6509487628936768, 1.6727478504180908, 1.7176315784454346, 1.6853704452514648, 1.6522839069366455, 1.6516473293304443, 1.7054896354675293, 1.6439743041992188, 1.725287914276123, 1.723382830619812, 1.6928086280822754, 1.699299931526184, 1.682389497756958, 1.652985692024231, 1.6361443996429443, 1.692456603050232, 1.758253574371338, 1.7123968601226807, 1.7566211223602295, 1.6904276609420776, 1.685346007347107, 1.6648344993591309, 1.617163062095642, 1.6886508464813232, 1.6757745742797852, 1.7696328163146973]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3949, "test_loss": 1.734569482421875}, {"step": 1000, "epoch": 2, "train_loss": 1.7891491651535034, "cos_r_negg": [0.7481253147125244, 0.08723568171262741, 0.1262476146221161, 0.14220815896987915, 0.11993048340082169, 0.21695870161056519, 0.12511128187179565, 0.10845031589269638, 0.13913270831108093, 0.1772518754005432, 0.0914146676659584, 0.13578735291957855, 0.11686636507511139, 0.10719402134418488, 0.14507591724395752, 0.13366393744945526, 0.09397129714488983, 0.16090305149555206, 0.10955445468425751, 0.08823902904987335, 0.10307028889656067, 0.13423550128936768, 0.1775968372821808, 0.15186253190040588, 0.12371474504470825, 0.15490469336509705, 0.18897703289985657, 0.13136544823646545, 0.11945229768753052, 0.1237349659204483], "cos_apical_negg": [0.7481253147125244, 0.08723568171262741, 0.1262476146221161, 0.14220815896987915, 0.11993048340082169, 0.21695870161056519, 0.12511128187179565, 0.10845031589269638, 0.13913270831108093, 0.1772518754005432, 0.0914146676659584, 0.13578735291957855, 0.11686636507511139, 0.10719402134418488, 0.14507591724395752, 0.13366393744945526, 0.09397129714488983, 0.16090305149555206, 0.10955445468425751, 0.08823902904987335, 0.10307028889656067, 0.13423550128936768, 0.1775968372821808, 0.15186253190040588, 0.12371474504470825, 0.15490469336509705, 0.18897703289985657, 0.13136544823646545, 0.11945229768753052, 0.1237349659204483], "cos_Ac_negg": [0.7481253147125244, 0.08723568171262741, 0.1262476146221161, 0.14220815896987915, 0.11993048340082169, 0.21695870161056519, 0.12511128187179565, 0.10845031589269638, 0.13913270831108093, 0.1772518754005432, 0.0914146676659584, 0.13578735291957855, 0.11686636507511139, 0.10719402134418488, 0.14507591724395752, 0.13366393744945526, 0.09397129714488983, 0.16090305149555206, 0.10955445468425751, 0.08823902904987335, 0.10307028889656067, 0.13423550128936768, 0.1775968372821808, 0.15186253190040588, 0.12371474504470825, 0.15490469336509705, 0.18897703289985657, 0.13136544823646545, 0.11945229768753052, 0.1237349659204483], "r_norm": [1.6140035390853882, 1.4254282712936401, 1.352888822555542, 1.467198133468628, 1.3718574047088623, 1.4000498056411743, 1.4238755702972412, 1.4012216329574585, 1.382458209991455, 1.3784303665161133, 1.406322717666626, 1.3642330169677734, 1.4337158203125, 1.4325206279754639, 1.4014835357666016, 1.411428451538086, 1.3939648866653442, 1.3725155591964722, 1.351965308189392, 1.402137279510498, 1.4632196426391602, 1.4216939210891724, 1.463111400604248, 1.4099078178405762, 1.391545295715332, 1.3863656520843506, 1.3543344736099243, 1.3989393711090088, 1.3938698768615723, 1.4669108390808105]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4022, "test_loss": 1.7277428100585936}, {"step": 1500, "epoch": 3, "train_loss": 1.6306061744689941, "cos_r_negg": [0.7680776119232178, 0.1833644062280655, 0.22911831736564636, 0.25512033700942993, 0.21580298244953156, 0.2717164158821106, 0.20507168769836426, 0.20333462953567505, 0.24219411611557007, 0.2623414695262909, 0.19906587898731232, 0.22251316905021667, 0.20990890264511108, 0.25363433361053467, 0.23657089471817017, 0.249003604054451, 0.17977386713027954, 0.2321949154138565, 0.22791868448257446, 0.19502292573451996, 0.20775309205055237, 0.21544554829597473, 0.2541475296020508, 0.25484105944633484, 0.21541228890419006, 0.28910529613494873, 0.30082273483276367, 0.22240614891052246, 0.25360873341560364, 0.20813892781734467], "cos_apical_negg": [0.7680776119232178, 0.1833644062280655, 0.22911831736564636, 0.25512033700942993, 0.21580298244953156, 0.2717164158821106, 0.20507168769836426, 0.20333462953567505, 0.24219411611557007, 0.2623414695262909, 0.19906587898731232, 0.22251316905021667, 0.20990890264511108, 0.25363433361053467, 0.23657089471817017, 0.249003604054451, 0.17977386713027954, 0.2321949154138565, 0.22791868448257446, 0.19502292573451996, 0.20775309205055237, 0.21544554829597473, 0.2541475296020508, 0.25484105944633484, 0.21541228890419006, 0.28910529613494873, 0.30082273483276367, 0.22240614891052246, 0.25360873341560364, 0.20813892781734467], "cos_Ac_negg": [0.7680776119232178, 0.1833644062280655, 0.22911831736564636, 0.25512033700942993, 0.21580298244953156, 0.2717164158821106, 0.20507168769836426, 0.20333462953567505, 0.24219411611557007, 0.2623414695262909, 0.19906587898731232, 0.22251316905021667, 0.20990890264511108, 0.25363433361053467, 0.23657089471817017, 0.249003604054451, 0.17977386713027954, 0.2321949154138565, 0.22791868448257446, 0.19502292573451996, 0.20775309205055237, 0.21544554829597473, 0.2541475296020508, 0.25484105944633484, 0.21541228890419006, 0.28910529613494873, 0.30082273483276367, 0.22240614891052246, 0.25360873341560364, 0.20813892781734467], "r_norm": [1.3953982591629028, 1.1564154624938965, 1.1046621799468994, 1.1938965320587158, 1.1198711395263672, 1.1520767211914062, 1.1528303623199463, 1.1342716217041016, 1.128793716430664, 1.1276687383651733, 1.138451337814331, 1.116047978401184, 1.1624228954315186, 1.1623008251190186, 1.1444140672683716, 1.1481754779815674, 1.1366971731185913, 1.1214282512664795, 1.1019845008850098, 1.1311326026916504, 1.182726263999939, 1.1570005416870117, 1.1968952417373657, 1.145625352859497, 1.1311750411987305, 1.134714126586914, 1.1178967952728271, 1.1285496950149536, 1.1378092765808105, 1.188349962234497]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4157, "test_loss": 1.7077855590820312}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4133, "test_loss": 1.6851546752929687}], "final": {"test_acc": 0.4133, "test_loss": 1.6851546752929687, "wall_s": 106.02875471115112, "cos_r_negg": [0.8100559115409851, 0.27810901403427124, 0.34168511629104614, 0.3301934599876404, 0.30952197313308716, 0.3673052489757538, 0.30103176832199097, 0.3039827346801758, 0.3352390229701996, 0.355302631855011, 0.29407763481140137, 0.3101213574409485, 0.2953321933746338, 0.34929782152175903, 0.32344403862953186, 0.35745474696159363, 0.29916155338287354, 0.32550105452537537, 0.3183419108390808, 0.3149386942386627, 0.3311288356781006, 0.32061469554901123, 0.3560485541820526, 0.3460298180580139, 0.32820266485214233, 0.4084368050098419, 0.4217104911804199, 0.3283073306083679, 0.3517517149448395, 0.3352779150009155], "cos_apical_negg": [0.8100559115409851, 0.27810901403427124, 0.34168511629104614, 0.3301934599876404, 0.30952197313308716, 0.3673052489757538, 0.30103176832199097, 0.3039827346801758, 0.3352390229701996, 0.355302631855011, 0.29407763481140137, 0.3101213574409485, 0.2953321933746338, 0.34929782152175903, 0.32344403862953186, 0.35745474696159363, 0.29916155338287354, 0.32550105452537537, 0.3183419108390808, 0.3149386942386627, 0.3311288356781006, 0.32061469554901123, 0.3560485541820526, 0.3460298180580139, 0.32820266485214233, 0.4084368050098419, 0.4217104911804199, 0.3283073306083679, 0.3517517149448395, 0.3352779150009155], "cos_Ac_negg": [0.8100559115409851, 0.27810901403427124, 0.34168511629104614, 0.3301934599876404, 0.30952197313308716, 0.3673052489757538, 0.30103176832199097, 0.3039827346801758, 0.3352390229701996, 0.355302631855011, 0.29407763481140137, 0.3101213574409485, 0.2953321933746338, 0.34929782152175903, 0.32344403862953186, 0.35745474696159363, 0.29916155338287354, 0.32550105452537537, 0.3183419108390808, 0.3149386942386627, 0.3311288356781006, 0.32061469554901123, 0.3560485541820526, 0.3460298180580139, 0.32820266485214233, 0.4084368050098419, 0.4217104911804199, 0.3283073306083679, 0.3517517149448395, 0.3352779150009155]}}
\ No newline at end of file diff --git a/results/pilot_sim_cifar10_sdil_w64_d60_s0.json b/results/pilot_sim_cifar10_sdil_w64_d60_s0.json new file mode 100644 index 0000000..9799123 --- /dev/null +++ b/results/pilot_sim_cifar10_sdil_w64_d60_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 60, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 16, "pert_mode": "simultaneous", "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_sim_cifar10_sdil_w64_d60_s0", "n_in": 3072}, "provenance": {"git_commit": "0f107d1ee4e750421a0e95f8bd06bf0f629d2fa7", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.6154470443725586, "cos_r_negg": [0.008188098669052124, 0.03132791817188263, -0.04745005816221237, -0.002227875404059887, -0.013256456702947617, -0.014939205721020699, -0.07091359794139862, -0.030459068715572357, -0.020020881667733192, -0.051334332674741745, 0.04942387342453003, -0.020939603447914124, 0.002700087148696184, -0.05190502479672432, -0.056835681200027466, -0.06955941021442413, 0.08165355026721954, 0.041968271136283875, 0.0524429976940155, -0.03258373588323593, 0.02309548109769821, -0.008639009669423103, 0.07035446166992188, 0.01823953166604042, -0.00805345643311739, -0.011262137442827225, -0.06306220591068268, -0.004426212981343269, -0.06595856696367264, -0.030616607517004013, 0.0438714325428009, 0.0039827860891819, -0.02320013754069805, -0.03326372057199478, 0.003818083554506302, 0.020670447498559952, -0.00602892367169261, 0.00011380738578736782, 0.030327288433909416, 0.016832441091537476, 0.09271088242530823, 0.020278532058000565, -0.012991376221179962, 0.037845999002456665, -0.04402860999107361, 0.0067482297308743, -0.04700296372175217, -0.0008557961555197835, -0.05770082771778107, 0.04755961894989014, 0.004939173348248005, -0.10112802684307098, 0.030835676938295364, 0.029946893453598022, -0.015894554555416107, 0.04179966822266579, 0.013654191978275776, -0.06056639924645424, -0.04872291162610054, -0.03112165257334709], "cos_apical_negg": [0.008188098669052124, 0.03132791817188263, -0.04745005816221237, -0.002227875404059887, -0.013256456702947617, -0.014939205721020699, -0.07091359794139862, -0.030459068715572357, -0.020020881667733192, -0.051334332674741745, 0.04942387342453003, -0.020939603447914124, 0.002700087148696184, -0.05190502479672432, -0.056835681200027466, -0.06955941021442413, 0.08165355026721954, 0.041968271136283875, 0.0524429976940155, -0.03258373588323593, 0.02309548109769821, -0.008639009669423103, 0.07035446166992188, 0.01823953166604042, -0.00805345643311739, -0.011262137442827225, -0.06306220591068268, -0.004426212981343269, -0.06595856696367264, -0.030616607517004013, 0.0438714325428009, 0.0039827860891819, -0.02320013754069805, -0.03326372057199478, 0.003818083554506302, 0.020670447498559952, -0.00602892367169261, 0.00011380738578736782, 0.030327288433909416, 0.016832441091537476, 0.09271088242530823, 0.020278532058000565, -0.012991376221179962, 0.037845999002456665, -0.04402860999107361, 0.0067482297308743, -0.04700296372175217, -0.0008557961555197835, -0.05770082771778107, 0.04755961894989014, 0.004939173348248005, -0.10112802684307098, 0.030835676938295364, 0.029946893453598022, -0.015894554555416107, 0.04179966822266579, 0.013654191978275776, -0.06056639924645424, -0.04872291162610054, -0.03112165257334709], "cos_Ac_negg": [0.008188098669052124, 0.03132791817188263, -0.04745005816221237, -0.002227875404059887, -0.013256456702947617, -0.014939205721020699, -0.07091359794139862, -0.030459068715572357, -0.020020881667733192, -0.051334332674741745, 0.04942387342453003, -0.020939603447914124, 0.002700087148696184, -0.05190502479672432, -0.056835681200027466, -0.06955941021442413, 0.08165355026721954, 0.041968271136283875, 0.0524429976940155, -0.03258373588323593, 0.02309548109769821, -0.008639009669423103, 0.07035446166992188, 0.01823953166604042, -0.00805345643311739, -0.011262137442827225, -0.06306220591068268, -0.004426212981343269, -0.06595856696367264, -0.030616607517004013, 0.0438714325428009, 0.0039827860891819, -0.02320013754069805, -0.03326372057199478, 0.003818083554506302, 0.020670447498559952, -0.00602892367169261, 0.00011380738578736782, 0.030327288433909416, 0.016832441091537476, 0.09271088242530823, 0.020278532058000565, -0.012991376221179962, 0.037845999002456665, -0.04402860999107361, 0.0067482297308743, -0.04700296372175217, -0.0008557961555197835, -0.05770082771778107, 0.04755961894989014, 0.004939173348248005, -0.10112802684307098, 0.030835676938295364, 0.029946893453598022, -0.015894554555416107, 0.04179966822266579, 0.013654191978275776, -0.06056639924645424, -0.04872291162610054, -0.03112165257334709], "r_norm": [2.475722312927246, 2.581672191619873, 2.5104451179504395, 2.4316773414611816, 2.2790603637695312, 2.4673492908477783, 2.376934766769409, 2.434628486633301, 2.592359781265259, 2.5058107376098633, 2.4041948318481445, 2.5822935104370117, 2.4772605895996094, 2.4681668281555176, 2.6032633781433105, 2.5102105140686035, 2.5783021450042725, 2.5189881324768066, 2.487623691558838, 2.527086019515991, 2.6228270530700684, 2.425225257873535, 2.425752639770508, 2.5075631141662598, 2.4634971618652344, 2.42893123626709, 2.6356470584869385, 2.4488370418548584, 2.370922088623047, 2.640894651412964, 2.3422160148620605, 2.447188377380371, 2.4530675411224365, 2.4734878540039062, 2.5735106468200684, 2.523186683654785, 2.401862621307373, 2.3840420246124268, 2.5408825874328613, 2.4573960304260254, 2.5272536277770996, 2.433478832244873, 2.4793224334716797, 2.4267563819885254, 2.510546922683716, 2.5136985778808594, 2.5523428916931152, 2.4561514854431152, 2.4074058532714844, 2.429624557495117, 2.395200252532959, 2.547451972961426, 2.3779640197753906, 2.4384965896606445, 2.555941581726074, 2.398407459259033, 2.6079585552215576, 2.485663890838623, 2.4005508422851562, 2.4267616271972656], "ldr": {"dL_sdil": 0.015146255493164062, "dL_bp": -0.4264340400695801, "ratio": -0.03551840160483598}}, {"epoch_end": 0, "step": 391, "test_acc": 0.3619, "test_loss": 1.80926533203125}, {"step": 500, "epoch": 1, "train_loss": 1.6660630702972412, "cos_r_negg": [0.6112304925918579, 0.09000800549983978, 0.044928938150405884, 0.022784523665905, 0.030601579695940018, 0.05130467563867569, 0.05879845470190048, 0.10867206007242203, 0.040181964635849, 0.004886255599558353, 0.07790626585483551, 0.02760295197367668, 0.044667042791843414, 0.021306147798895836, 0.09223973751068115, 0.026670049875974655, 0.043648310005664825, 0.056379079818725586, 0.09735850989818573, 0.07787682116031647, 0.04671887308359146, 0.09523791074752808, 0.1159275695681572, 0.09864035248756409, 0.14201515913009644, 0.06158149242401123, 0.04996059089899063, -0.03743249922990799, 0.042612869292497635, -0.007639927789568901, 0.10276351869106293, 0.07502959668636322, 0.0053717647679150105, 0.07206674665212631, 0.06370322406291962, 0.10443229973316193, 0.029140260070562363, 0.019112512469291687, 0.04475380852818489, 0.018761005252599716, 0.10706733167171478, 0.11353138089179993, 0.016565771773457527, 0.06804738938808441, 0.037905264645814896, 0.04407908022403717, 0.04533611610531807, 0.13673312962055206, -0.015630923211574554, 0.025380907580256462, -0.021598290652036667, -0.016693541780114174, 0.07548674941062927, 0.12835091352462769, 0.14071142673492432, 0.06848399341106415, 0.07631034404039383, 0.09373270720243454, -0.09031413495540619, 0.011114958673715591], "cos_apical_negg": [0.6112304925918579, 0.09000800549983978, 0.044928938150405884, 0.022784523665905, 0.030601579695940018, 0.05130467563867569, 0.05879845470190048, 0.10867206007242203, 0.040181964635849, 0.004886255599558353, 0.07790626585483551, 0.02760295197367668, 0.044667042791843414, 0.021306147798895836, 0.09223973751068115, 0.026670049875974655, 0.043648310005664825, 0.056379079818725586, 0.09735850989818573, 0.07787682116031647, 0.04671887308359146, 0.09523791074752808, 0.1159275695681572, 0.09864035248756409, 0.14201515913009644, 0.06158149242401123, 0.04996059089899063, -0.03743249922990799, 0.042612869292497635, -0.007639927789568901, 0.10276351869106293, 0.07502959668636322, 0.0053717647679150105, 0.07206674665212631, 0.06370322406291962, 0.10443229973316193, 0.029140260070562363, 0.019112512469291687, 0.04475380852818489, 0.018761005252599716, 0.10706733167171478, 0.11353138089179993, 0.016565771773457527, 0.06804738938808441, 0.037905264645814896, 0.04407908022403717, 0.04533611610531807, 0.13673312962055206, -0.015630923211574554, 0.025380907580256462, -0.021598290652036667, -0.016693541780114174, 0.07548674941062927, 0.12835091352462769, 0.14071142673492432, 0.06848399341106415, 0.07631034404039383, 0.09373270720243454, -0.09031413495540619, 0.011114958673715591], "cos_Ac_negg": [0.6112304925918579, 0.09000800549983978, 0.044928938150405884, 0.022784523665905, 0.030601579695940018, 0.05130467563867569, 0.05879845470190048, 0.10867206007242203, 0.040181964635849, 0.004886255599558353, 0.07790626585483551, 0.02760295197367668, 0.044667042791843414, 0.021306147798895836, 0.09223973751068115, 0.026670049875974655, 0.043648310005664825, 0.056379079818725586, 0.09735850989818573, 0.07787682116031647, 0.04671887308359146, 0.09523791074752808, 0.1159275695681572, 0.09864035248756409, 0.14201515913009644, 0.06158149242401123, 0.04996059089899063, -0.03743249922990799, 0.042612869292497635, -0.007639927789568901, 0.10276351869106293, 0.07502959668636322, 0.0053717647679150105, 0.07206674665212631, 0.06370322406291962, 0.10443229973316193, 0.029140260070562363, 0.019112512469291687, 0.04475380852818489, 0.018761005252599716, 0.10706733167171478, 0.11353138089179993, 0.016565771773457527, 0.06804738938808441, 0.037905264645814896, 0.04407908022403717, 0.04533611610531807, 0.13673312962055206, -0.015630923211574554, 0.025380907580256462, -0.021598290652036667, -0.016693541780114174, 0.07548674941062927, 0.12835091352462769, 0.14071142673492432, 0.06848399341106415, 0.07631034404039383, 0.09373270720243454, -0.09031413495540619, 0.011114958673715591], "r_norm": [1.7520763874053955, 1.766920566558838, 1.7303932905197144, 1.6763145923614502, 1.5762789249420166, 1.6812564134597778, 1.616024374961853, 1.6734426021575928, 1.7926931381225586, 1.713576316833496, 1.672144889831543, 1.753313422203064, 1.6941341161727905, 1.6868164539337158, 1.7898908853530884, 1.7138323783874512, 1.7655799388885498, 1.709261417388916, 1.7114425897598267, 1.7462739944458008, 1.8087623119354248, 1.6744320392608643, 1.6850876808166504, 1.7198946475982666, 1.7012033462524414, 1.6709038019180298, 1.8157541751861572, 1.6835846900939941, 1.6287428140640259, 1.8034865856170654, 1.6017817258834839, 1.6871418952941895, 1.6592463254928589, 1.7022277116775513, 1.7667028903961182, 1.7495498657226562, 1.640779972076416, 1.6459012031555176, 1.7476775646209717, 1.6883957386016846, 1.7556278705596924, 1.6741769313812256, 1.7022665739059448, 1.6788749694824219, 1.7135440111160278, 1.7291560173034668, 1.7452316284179688, 1.7010447978973389, 1.6530239582061768, 1.6732720136642456, 1.6451759338378906, 1.7638916969299316, 1.6559914350509644, 1.6842113733291626, 1.7604870796203613, 1.6597012281417847, 1.7804629802703857, 1.7029420137405396, 1.6510965824127197, 1.6611117124557495]}, {"epoch_end": 1, "step": 782, "test_acc": 0.392, "test_loss": 1.75426728515625}, {"step": 1000, "epoch": 2, "train_loss": 1.7955200672149658, "cos_r_negg": [0.7283141016960144, 0.12397865951061249, 0.07722286880016327, 0.0990663468837738, 0.10357300937175751, 0.1530320644378662, 0.13637344539165497, 0.15010394155979156, 0.11963604390621185, 0.11117586493492126, 0.1509389579296112, 0.05456150695681572, 0.12177175283432007, 0.11703399568796158, 0.1454901099205017, 0.044380199164152145, 0.07652189582586288, 0.11796985566616058, 0.15015453100204468, 0.17564058303833008, 0.1257973462343216, 0.15391501784324646, 0.14797170460224152, 0.23963387310504913, 0.20538128912448883, 0.18966582417488098, 0.10512446612119675, 0.037836551666259766, 0.11386585235595703, 0.062348999083042145, 0.2124868631362915, 0.15442901849746704, 0.08915965259075165, 0.10933101177215576, 0.13747796416282654, 0.15771083533763885, 0.1466752588748932, 0.1096687987446785, 0.09196659922599792, 0.10895353555679321, 0.20359939336776733, 0.16457532346248627, 0.09495964646339417, 0.1459512710571289, 0.0896441787481308, 0.14316484332084656, 0.15017956495285034, 0.17479172348976135, 0.0769026130437851, 0.09786699712276459, 0.03322369232773781, 0.08382532000541687, 0.10217329859733582, 0.18325230479240417, 0.21581311523914337, 0.15000763535499573, 0.13540419936180115, 0.19928213953971863, 0.06684704124927521, 0.08040070533752441], "cos_apical_negg": [0.7283141016960144, 0.12397865951061249, 0.07722286880016327, 0.0990663468837738, 0.10357300937175751, 0.1530320644378662, 0.13637344539165497, 0.15010394155979156, 0.11963604390621185, 0.11117586493492126, 0.1509389579296112, 0.05456150695681572, 0.12177175283432007, 0.11703399568796158, 0.1454901099205017, 0.044380199164152145, 0.07652189582586288, 0.11796985566616058, 0.15015453100204468, 0.17564058303833008, 0.1257973462343216, 0.15391501784324646, 0.14797170460224152, 0.23963387310504913, 0.20538128912448883, 0.18966582417488098, 0.10512446612119675, 0.037836551666259766, 0.11386585235595703, 0.062348999083042145, 0.2124868631362915, 0.15442901849746704, 0.08915965259075165, 0.10933101177215576, 0.13747796416282654, 0.15771083533763885, 0.1466752588748932, 0.1096687987446785, 0.09196659922599792, 0.10895353555679321, 0.20359939336776733, 0.16457532346248627, 0.09495964646339417, 0.1459512710571289, 0.0896441787481308, 0.14316484332084656, 0.15017956495285034, 0.17479172348976135, 0.0769026130437851, 0.09786699712276459, 0.03322369232773781, 0.08382532000541687, 0.10217329859733582, 0.18325230479240417, 0.21581311523914337, 0.15000763535499573, 0.13540419936180115, 0.19928213953971863, 0.06684704124927521, 0.08040070533752441], "cos_Ac_negg": [0.7283141016960144, 0.12397865951061249, 0.07722286880016327, 0.0990663468837738, 0.10357300937175751, 0.1530320644378662, 0.13637344539165497, 0.15010394155979156, 0.11963604390621185, 0.11117586493492126, 0.1509389579296112, 0.05456150695681572, 0.12177175283432007, 0.11703399568796158, 0.1454901099205017, 0.044380199164152145, 0.07652189582586288, 0.11796985566616058, 0.15015453100204468, 0.17564058303833008, 0.1257973462343216, 0.15391501784324646, 0.14797170460224152, 0.23963387310504913, 0.20538128912448883, 0.18966582417488098, 0.10512446612119675, 0.037836551666259766, 0.11386585235595703, 0.062348999083042145, 0.2124868631362915, 0.15442901849746704, 0.08915965259075165, 0.10933101177215576, 0.13747796416282654, 0.15771083533763885, 0.1466752588748932, 0.1096687987446785, 0.09196659922599792, 0.10895353555679321, 0.20359939336776733, 0.16457532346248627, 0.09495964646339417, 0.1459512710571289, 0.0896441787481308, 0.14316484332084656, 0.15017956495285034, 0.17479172348976135, 0.0769026130437851, 0.09786699712276459, 0.03322369232773781, 0.08382532000541687, 0.10217329859733582, 0.18325230479240417, 0.21581311523914337, 0.15000763535499573, 0.13540419936180115, 0.19928213953971863, 0.06684704124927521, 0.08040070533752441], "r_norm": [1.5162702798843384, 1.4437369108200073, 1.4186245203018188, 1.3624285459518433, 1.2946631908416748, 1.3795342445373535, 1.3301594257354736, 1.3690296411514282, 1.474086880683899, 1.3982348442077637, 1.3665202856063843, 1.433239221572876, 1.3859946727752686, 1.378375768661499, 1.4720566272735596, 1.3937890529632568, 1.4350709915161133, 1.3990051746368408, 1.4112764596939087, 1.431164264678955, 1.4802019596099854, 1.371061086654663, 1.3806287050247192, 1.4106919765472412, 1.3990602493286133, 1.3737479448318481, 1.491297960281372, 1.3631949424743652, 1.3261017799377441, 1.4662944078445435, 1.320297360420227, 1.3802337646484375, 1.3585920333862305, 1.3867385387420654, 1.4451172351837158, 1.434313178062439, 1.3478941917419434, 1.3408222198486328, 1.4220380783081055, 1.385351538658142, 1.4433773756027222, 1.3722491264343262, 1.3892741203308105, 1.3717788457870483, 1.3982517719268799, 1.4182720184326172, 1.4286270141601562, 1.399727702140808, 1.3468252420425415, 1.3730473518371582, 1.3270926475524902, 1.440374493598938, 1.3557770252227783, 1.3819899559020996, 1.446204423904419, 1.3592655658721924, 1.4582908153533936, 1.3990848064422607, 1.3458908796310425, 1.3503236770629883]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4028, "test_loss": 1.7169515991210937}, {"step": 1500, "epoch": 3, "train_loss": 1.594815969467163, "cos_r_negg": [0.7841651439666748, 0.2161160409450531, 0.1961853951215744, 0.18456125259399414, 0.23227360844612122, 0.24110907316207886, 0.23738275468349457, 0.24352815747261047, 0.21226640045642853, 0.17797112464904785, 0.262307733297348, 0.17668496072292328, 0.19149498641490936, 0.21754074096679688, 0.24505764245986938, 0.1617370843887329, 0.17980742454528809, 0.2752155363559723, 0.2516050338745117, 0.2537948489189148, 0.217879980802536, 0.24332749843597412, 0.23424136638641357, 0.2852822244167328, 0.30759215354919434, 0.253506600856781, 0.16954253613948822, 0.11346258223056793, 0.24361751973628998, 0.15761670470237732, 0.30523914098739624, 0.2543443441390991, 0.16462290287017822, 0.19498077034950256, 0.22023391723632812, 0.2586563229560852, 0.25071412324905396, 0.21320423483848572, 0.19206826388835907, 0.204280823469162, 0.26202476024627686, 0.26420727372169495, 0.17556732892990112, 0.20808789134025574, 0.19919845461845398, 0.24238885939121246, 0.23006713390350342, 0.2808533310890198, 0.17946100234985352, 0.20193298161029816, 0.14356201887130737, 0.22060465812683105, 0.2116435021162033, 0.26824069023132324, 0.29073798656463623, 0.25586333870887756, 0.25466328859329224, 0.3074699938297272, 0.13738617300987244, 0.1778407096862793], "cos_apical_negg": [0.7841651439666748, 0.2161160409450531, 0.1961853951215744, 0.18456125259399414, 0.23227360844612122, 0.24110907316207886, 0.23738275468349457, 0.24352815747261047, 0.21226640045642853, 0.17797112464904785, 0.262307733297348, 0.17668496072292328, 0.19149498641490936, 0.21754074096679688, 0.24505764245986938, 0.1617370843887329, 0.17980742454528809, 0.2752155363559723, 0.2516050338745117, 0.2537948489189148, 0.217879980802536, 0.24332749843597412, 0.23424136638641357, 0.2852822244167328, 0.30759215354919434, 0.253506600856781, 0.16954253613948822, 0.11346258223056793, 0.24361751973628998, 0.15761670470237732, 0.30523914098739624, 0.2543443441390991, 0.16462290287017822, 0.19498077034950256, 0.22023391723632812, 0.2586563229560852, 0.25071412324905396, 0.21320423483848572, 0.19206826388835907, 0.204280823469162, 0.26202476024627686, 0.26420727372169495, 0.17556732892990112, 0.20808789134025574, 0.19919845461845398, 0.24238885939121246, 0.23006713390350342, 0.2808533310890198, 0.17946100234985352, 0.20193298161029816, 0.14356201887130737, 0.22060465812683105, 0.2116435021162033, 0.26824069023132324, 0.29073798656463623, 0.25586333870887756, 0.25466328859329224, 0.3074699938297272, 0.13738617300987244, 0.1778407096862793], "cos_Ac_negg": [0.7841651439666748, 0.2161160409450531, 0.1961853951215744, 0.18456125259399414, 0.23227360844612122, 0.24110907316207886, 0.23738275468349457, 0.24352815747261047, 0.21226640045642853, 0.17797112464904785, 0.262307733297348, 0.17668496072292328, 0.19149498641490936, 0.21754074096679688, 0.24505764245986938, 0.1617370843887329, 0.17980742454528809, 0.2752155363559723, 0.2516050338745117, 0.2537948489189148, 0.217879980802536, 0.24332749843597412, 0.23424136638641357, 0.2852822244167328, 0.30759215354919434, 0.253506600856781, 0.16954253613948822, 0.11346258223056793, 0.24361751973628998, 0.15761670470237732, 0.30523914098739624, 0.2543443441390991, 0.16462290287017822, 0.19498077034950256, 0.22023391723632812, 0.2586563229560852, 0.25071412324905396, 0.21320423483848572, 0.19206826388835907, 0.204280823469162, 0.26202476024627686, 0.26420727372169495, 0.17556732892990112, 0.20808789134025574, 0.19919845461845398, 0.24238885939121246, 0.23006713390350342, 0.2808533310890198, 0.17946100234985352, 0.20193298161029816, 0.14356201887130737, 0.22060465812683105, 0.2116435021162033, 0.26824069023132324, 0.29073798656463623, 0.25586333870887756, 0.25466328859329224, 0.3074699938297272, 0.13738617300987244, 0.1778407096862793], "r_norm": [1.3412169218063354, 1.1955029964447021, 1.169551134109497, 1.1279528141021729, 1.0767290592193604, 1.1497975587844849, 1.112419843673706, 1.1420049667358398, 1.2167010307312012, 1.1575387716293335, 1.1368540525436401, 1.1851744651794434, 1.1444880962371826, 1.139318823814392, 1.2291351556777954, 1.1380887031555176, 1.1862664222717285, 1.16620671749115, 1.172590970993042, 1.1854867935180664, 1.2166485786437988, 1.138096570968628, 1.1533963680267334, 1.1770051717758179, 1.1696887016296387, 1.1468814611434937, 1.233482003211975, 1.1201822757720947, 1.0976669788360596, 1.21329927444458, 1.1162488460540771, 1.1514217853546143, 1.1221919059753418, 1.1523878574371338, 1.1937062740325928, 1.1919960975646973, 1.1253868341445923, 1.1093482971191406, 1.1690483093261719, 1.1526436805725098, 1.2001460790634155, 1.1467821598052979, 1.1473064422607422, 1.1328951120376587, 1.1583421230316162, 1.1768639087677002, 1.181457281112671, 1.1651825904846191, 1.1113417148590088, 1.1236951351165771, 1.085038423538208, 1.1904107332229614, 1.1124787330627441, 1.1630470752716064, 1.2148313522338867, 1.129392385482788, 1.2163174152374268, 1.1718720197677612, 1.0985890626907349, 1.1132700443267822]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4075, "test_loss": 1.7215805053710938}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4083, "test_loss": 1.7022966186523438}], "final": {"test_acc": 0.4083, "test_loss": 1.7022966186523438, "wall_s": 213.1677463054657, "cos_r_negg": [0.7801978588104248, 0.3091685175895691, 0.31516051292419434, 0.29503870010375977, 0.3405274748802185, 0.33319759368896484, 0.30835410952568054, 0.3405994772911072, 0.2993738651275635, 0.2872163653373718, 0.36008089780807495, 0.30455726385116577, 0.34036552906036377, 0.3057326674461365, 0.3333376348018646, 0.24812127649784088, 0.26849642395973206, 0.35599270462989807, 0.3621777892112732, 0.3366605043411255, 0.3188316226005554, 0.33639609813690186, 0.3429409861564636, 0.38857609033584595, 0.394670695066452, 0.355517715215683, 0.2923544645309448, 0.2525053024291992, 0.32503175735473633, 0.23014283180236816, 0.37797093391418457, 0.3487055003643036, 0.27684497833251953, 0.30512434244155884, 0.3378211259841919, 0.3332935571670532, 0.35276827216148376, 0.3120586574077606, 0.28984498977661133, 0.3073764443397522, 0.3814936876296997, 0.3590427041053772, 0.28722456097602844, 0.28970253467559814, 0.33874714374542236, 0.35058075189590454, 0.3473470211029053, 0.3788111209869385, 0.3193284571170807, 0.31394970417022705, 0.28791821002960205, 0.2856176495552063, 0.3209686279296875, 0.3656827211380005, 0.3633660674095154, 0.3561963140964508, 0.3391960561275482, 0.41389453411102295, 0.256534218788147, 0.2908320426940918], "cos_apical_negg": [0.7801978588104248, 0.3091685175895691, 0.31516051292419434, 0.29503870010375977, 0.3405274748802185, 0.33319759368896484, 0.30835410952568054, 0.3405994772911072, 0.2993738651275635, 0.2872163653373718, 0.36008089780807495, 0.30455726385116577, 0.34036552906036377, 0.3057326674461365, 0.3333376348018646, 0.24812127649784088, 0.26849642395973206, 0.35599270462989807, 0.3621777892112732, 0.3366605043411255, 0.3188316226005554, 0.33639609813690186, 0.3429409861564636, 0.38857609033584595, 0.394670695066452, 0.355517715215683, 0.2923544645309448, 0.2525053024291992, 0.32503175735473633, 0.23014283180236816, 0.37797093391418457, 0.3487055003643036, 0.27684497833251953, 0.30512434244155884, 0.3378211259841919, 0.3332935571670532, 0.35276827216148376, 0.3120586574077606, 0.28984498977661133, 0.3073764443397522, 0.3814936876296997, 0.3590427041053772, 0.28722456097602844, 0.28970253467559814, 0.33874714374542236, 0.35058075189590454, 0.3473470211029053, 0.3788111209869385, 0.3193284571170807, 0.31394970417022705, 0.28791821002960205, 0.2856176495552063, 0.3209686279296875, 0.3656827211380005, 0.3633660674095154, 0.3561963140964508, 0.3391960561275482, 0.41389453411102295, 0.256534218788147, 0.2908320426940918], "cos_Ac_negg": [0.7801978588104248, 0.3091685175895691, 0.31516051292419434, 0.29503870010375977, 0.3405274748802185, 0.33319759368896484, 0.30835410952568054, 0.3405994772911072, 0.2993738651275635, 0.2872163653373718, 0.36008089780807495, 0.30455726385116577, 0.34036552906036377, 0.3057326674461365, 0.3333376348018646, 0.24812127649784088, 0.26849642395973206, 0.35599270462989807, 0.3621777892112732, 0.3366605043411255, 0.3188316226005554, 0.33639609813690186, 0.3429409861564636, 0.38857609033584595, 0.394670695066452, 0.355517715215683, 0.2923544645309448, 0.2525053024291992, 0.32503175735473633, 0.23014283180236816, 0.37797093391418457, 0.3487055003643036, 0.27684497833251953, 0.30512434244155884, 0.3378211259841919, 0.3332935571670532, 0.35276827216148376, 0.3120586574077606, 0.28984498977661133, 0.3073764443397522, 0.3814936876296997, 0.3590427041053772, 0.28722456097602844, 0.28970253467559814, 0.33874714374542236, 0.35058075189590454, 0.3473470211029053, 0.3788111209869385, 0.3193284571170807, 0.31394970417022705, 0.28791821002960205, 0.2856176495552063, 0.3209686279296875, 0.3656827211380005, 0.3633660674095154, 0.3561963140964508, 0.3391960561275482, 0.41389453411102295, 0.256534218788147, 0.2908320426940918]}}
\ No newline at end of file diff --git a/results/pilot_simvec_cifar10_sdil_w64_d30_s0.json b/results/pilot_simvec_cifar10_sdil_w64_d30_s0.json new file mode 100644 index 0000000..7cd8dd2 --- /dev/null +++ b/results/pilot_simvec_cifar10_sdil_w64_d30_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 30, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 16, "pert_mode": "simultaneous", "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_simvec_cifar10_sdil_w64_d30_s0", "n_in": 3072}, "provenance": {"git_commit": "e0bfdfd51c78b945242bfa36a485d8773c4e17bb", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.4157285690307617, "cos_r_negg": [0.007712585851550102, -0.03975548595190048, 0.025203227996826172, 0.03284638747572899, -0.02114245295524597, -0.02241712063550949, 0.01855619251728058, -0.011118985712528229, 0.09588839113712311, -0.023850148543715477, -0.046825066208839417, 0.003365863114595413, 0.0047249626368284225, 0.023461898788809776, -0.01765047386288643, 0.03156798332929611, -0.01771402545273304, 0.004477180540561676, 0.0037541198544204235, 0.035115797072649, 0.04352542757987976, -0.009780683554708958, 0.034562088549137115, -0.032053593546152115, -0.05173061788082123, -0.044041719287633896, 0.01968034915626049, 0.007763845846056938, 0.05400868505239487, 0.04457078129053116], "cos_apical_negg": [0.007712585851550102, -0.03975548595190048, 0.025203227996826172, 0.03284638747572899, -0.02114245295524597, -0.02241712063550949, 0.01855619251728058, -0.011118985712528229, 0.09588839113712311, -0.023850148543715477, -0.046825066208839417, 0.003365863114595413, 0.0047249626368284225, 0.023461898788809776, -0.01765047386288643, 0.03156798332929611, -0.01771402545273304, 0.004477180540561676, 0.0037541198544204235, 0.035115797072649, 0.04352542757987976, -0.009780683554708958, 0.034562088549137115, -0.032053593546152115, -0.05173061788082123, -0.044041719287633896, 0.01968034915626049, 0.007763845846056938, 0.05400868505239487, 0.04457078129053116], "cos_Ac_negg": [0.007712585851550102, -0.03975548595190048, 0.025203227996826172, 0.03284638747572899, -0.02114245295524597, -0.02241712063550949, 0.01855619251728058, -0.011118985712528229, 0.09588839113712311, -0.023850148543715477, -0.046825066208839417, 0.003365863114595413, 0.0047249626368284225, 0.023461898788809776, -0.01765047386288643, 0.03156798332929611, -0.01771402545273304, 0.004477180540561676, 0.0037541198544204235, 0.035115797072649, 0.04352542757987976, -0.009780683554708958, 0.034562088549137115, -0.032053593546152115, -0.05173061788082123, -0.044041719287633896, 0.01968034915626049, 0.007763845846056938, 0.05400868505239487, 0.04457078129053116], "r_norm": [2.585965156555176, 2.487182140350342, 2.3580644130706787, 2.535940170288086, 2.390333652496338, 2.4134573936462402, 2.5035479068756104, 2.4554741382598877, 2.377804756164551, 2.3752760887145996, 2.4523282051086426, 2.3806276321411133, 2.4788317680358887, 2.501803398132324, 2.442944049835205, 2.4293007850646973, 2.429853916168213, 2.3894612789154053, 2.3824052810668945, 2.4184999465942383, 2.5621697902679443, 2.4488232135772705, 2.551830530166626, 2.4550294876098633, 2.407695770263672, 2.4057023525238037, 2.3515889644622803, 2.39849853515625, 2.419497489929199, 2.563612461090088], "ldr": {"dL_sdil": 0.0012927055358886719, "dL_bp": -0.35697102546691895, "ratio": -0.0036213178203967954}}, {"epoch_end": 0, "step": 391, "test_acc": 0.3566, "test_loss": 1.8165755615234376}, {"step": 500, "epoch": 1, "train_loss": 1.762270450592041, "cos_r_negg": [0.6558809280395508, 0.07066728174686432, 0.018251970410346985, 0.08423072099685669, 0.014873884618282318, 0.09653906524181366, 0.06056039035320282, 0.014749763533473015, 0.11072409898042679, 0.10980172455310822, 0.06773938983678818, 0.06826259195804596, 0.055884283035993576, 0.017878936603665352, 0.058886896818876266, 0.09598827362060547, 0.03144843131303787, 0.03407925367355347, 0.05765664204955101, 0.008012301288545132, 0.04529644548892975, 0.05381857231259346, 0.11304923892021179, 0.08175028860569, 0.011697675101459026, 0.10300730168819427, 0.11308272182941437, 0.03464073687791824, 0.06281882524490356, 0.04331579804420471], "cos_apical_negg": [0.6558809280395508, 0.07066728174686432, 0.018251970410346985, 0.08423072099685669, 0.014873884618282318, 0.09653906524181366, 0.06056039035320282, 0.014749763533473015, 0.11072409898042679, 0.10980172455310822, 0.06773938983678818, 0.06826259195804596, 0.055884283035993576, 0.017878936603665352, 0.058886896818876266, 0.09598827362060547, 0.03144843131303787, 0.03407925367355347, 0.05765664204955101, 0.008012301288545132, 0.04529644548892975, 0.05381857231259346, 0.11304923892021179, 0.08175028860569, 0.011697675101459026, 0.10300730168819427, 0.11308272182941437, 0.03464073687791824, 0.06281882524490356, 0.04331579804420471], "cos_Ac_negg": [0.6558809280395508, 0.07066728174686432, 0.018251970410346985, 0.08423072099685669, 0.014873884618282318, 0.09653906524181366, 0.06056039035320282, 0.014749763533473015, 0.11072409898042679, 0.10980172455310822, 0.06773938983678818, 0.06826259195804596, 0.055884283035993576, 0.017878936603665352, 0.058886896818876266, 0.09598827362060547, 0.03144843131303787, 0.03407925367355347, 0.05765664204955101, 0.008012301288545132, 0.04529644548892975, 0.05381857231259346, 0.11304923892021179, 0.08175028860569, 0.011697675101459026, 0.10300730168819427, 0.11308272182941437, 0.03464073687791824, 0.06281882524490356, 0.04331579804420471], "r_norm": [1.8839044570922852, 1.7546790838241577, 1.665805697441101, 1.7955719232559204, 1.6785086393356323, 1.69894540309906, 1.7422223091125488, 1.721347451210022, 1.680301308631897, 1.6831727027893066, 1.7355031967163086, 1.6772198677062988, 1.762723445892334, 1.753968358039856, 1.7252929210662842, 1.7384850978851318, 1.716105580329895, 1.6800692081451416, 1.6614333391189575, 1.717505931854248, 1.781955599784851, 1.7441153526306152, 1.7911173105239868, 1.7119789123535156, 1.7116410732269287, 1.6895153522491455, 1.6457180976867676, 1.7111082077026367, 1.708539605140686, 1.8005788326263428]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3955, "test_loss": 1.745744482421875}, {"step": 1000, "epoch": 2, "train_loss": 1.677492380142212, "cos_r_negg": [0.7577265501022339, 0.1385941356420517, 0.11794496327638626, 0.1278281807899475, 0.06521080434322357, 0.23107996582984924, 0.14009007811546326, 0.09118485450744629, 0.1453491449356079, 0.15269191563129425, 0.12324953824281693, 0.16506412625312805, 0.13278870284557343, 0.11645059287548065, 0.14895015954971313, 0.15800440311431885, 0.06712011247873306, 0.08423195779323578, 0.1097240224480629, 0.07295048236846924, 0.08608043938875198, 0.1420779824256897, 0.2018738090991974, 0.1525341272354126, 0.13951443135738373, 0.16743025183677673, 0.19073817133903503, 0.11484451591968536, 0.12993669509887695, 0.14408499002456665], "cos_apical_negg": [0.7577265501022339, 0.1385941356420517, 0.11794496327638626, 0.1278281807899475, 0.06521080434322357, 0.23107996582984924, 0.14009007811546326, 0.09118485450744629, 0.1453491449356079, 0.15269191563129425, 0.12324953824281693, 0.16506412625312805, 0.13278870284557343, 0.11645059287548065, 0.14895015954971313, 0.15800440311431885, 0.06712011247873306, 0.08423195779323578, 0.1097240224480629, 0.07295048236846924, 0.08608043938875198, 0.1420779824256897, 0.2018738090991974, 0.1525341272354126, 0.13951443135738373, 0.16743025183677673, 0.19073817133903503, 0.11484451591968536, 0.12993669509887695, 0.14408499002456665], "cos_Ac_negg": [0.7577265501022339, 0.1385941356420517, 0.11794496327638626, 0.1278281807899475, 0.06521080434322357, 0.23107996582984924, 0.14009007811546326, 0.09118485450744629, 0.1453491449356079, 0.15269191563129425, 0.12324953824281693, 0.16506412625312805, 0.13278870284557343, 0.11645059287548065, 0.14895015954971313, 0.15800440311431885, 0.06712011247873306, 0.08423195779323578, 0.1097240224480629, 0.07295048236846924, 0.08608043938875198, 0.1420779824256897, 0.2018738090991974, 0.1525341272354126, 0.13951443135738373, 0.16743025183677673, 0.19073817133903503, 0.11484451591968536, 0.12993669509887695, 0.14408499002456665], "r_norm": [1.6054446697235107, 1.4141788482666016, 1.3471059799194336, 1.4502768516540527, 1.3544790744781494, 1.3885712623596191, 1.4123278856277466, 1.3809483051300049, 1.3676725625991821, 1.3633958101272583, 1.4063048362731934, 1.3560385704040527, 1.417107105255127, 1.4133042097091675, 1.392833948135376, 1.4055331945419312, 1.3792095184326172, 1.3562562465667725, 1.340383768081665, 1.377598762512207, 1.4441884756088257, 1.4073444604873657, 1.453507900238037, 1.3987061977386475, 1.377488613128662, 1.3749264478683472, 1.3416244983673096, 1.3841962814331055, 1.3809940814971924, 1.4571963548660278]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4014, "test_loss": 1.721560546875}, {"step": 1500, "epoch": 3, "train_loss": 1.6506505012512207, "cos_r_negg": [0.7669909000396729, 0.2235923707485199, 0.201499342918396, 0.2277807593345642, 0.1550195962190628, 0.30092760920524597, 0.20241017639636993, 0.2018587291240692, 0.2682421803474426, 0.2446775734424591, 0.19383856654167175, 0.25659239292144775, 0.1819555163383484, 0.22689968347549438, 0.2452574074268341, 0.22027890384197235, 0.1669909656047821, 0.20393279194831848, 0.2563086748123169, 0.18761980533599854, 0.19737735390663147, 0.22478537261486053, 0.27947884798049927, 0.24358010292053223, 0.24352812767028809, 0.25636184215545654, 0.3240711987018585, 0.26102638244628906, 0.2404536008834839, 0.24738478660583496], "cos_apical_negg": [0.7669909000396729, 0.2235923707485199, 0.201499342918396, 0.2277807593345642, 0.1550195962190628, 0.30092760920524597, 0.20241017639636993, 0.2018587291240692, 0.2682421803474426, 0.2446775734424591, 0.19383856654167175, 0.25659239292144775, 0.1819555163383484, 0.22689968347549438, 0.2452574074268341, 0.22027890384197235, 0.1669909656047821, 0.20393279194831848, 0.2563086748123169, 0.18761980533599854, 0.19737735390663147, 0.22478537261486053, 0.27947884798049927, 0.24358010292053223, 0.24352812767028809, 0.25636184215545654, 0.3240711987018585, 0.26102638244628906, 0.2404536008834839, 0.24738478660583496], "cos_Ac_negg": [0.7669909000396729, 0.2235923707485199, 0.201499342918396, 0.2277807593345642, 0.1550195962190628, 0.30092760920524597, 0.20241017639636993, 0.2018587291240692, 0.2682421803474426, 0.2446775734424591, 0.19383856654167175, 0.25659239292144775, 0.1819555163383484, 0.22689968347549438, 0.2452574074268341, 0.22027890384197235, 0.1669909656047821, 0.20393279194831848, 0.2563086748123169, 0.18761980533599854, 0.19737735390663147, 0.22478537261486053, 0.27947884798049927, 0.24358010292053223, 0.24352812767028809, 0.25636184215545654, 0.3240711987018585, 0.26102638244628906, 0.2404536008834839, 0.24738478660583496], "r_norm": [1.3774014711380005, 1.1432597637176514, 1.083369255065918, 1.175041913986206, 1.0901926755905151, 1.1368950605392456, 1.1416852474212646, 1.111236572265625, 1.1139365434646606, 1.1099488735198975, 1.138288974761963, 1.1012654304504395, 1.144324541091919, 1.1400047540664673, 1.1340140104293823, 1.1459838151931763, 1.11038076877594, 1.1021370887756348, 1.093240737915039, 1.1028854846954346, 1.1586533784866333, 1.1386052370071411, 1.1901698112487793, 1.1362440586090088, 1.1151857376098633, 1.1161550283432007, 1.100908637046814, 1.1157726049423218, 1.120802640914917, 1.1745201349258423]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4076, "test_loss": 1.697406640625}, {"epoch_end": 4, "step": 1955, "test_acc": 0.4146, "test_loss": 1.68923310546875}], "final": {"test_acc": 0.4146, "test_loss": 1.68923310546875, "wall_s": 47.33752679824829, "cos_r_negg": [0.7844280004501343, 0.2979603707790375, 0.30254095792770386, 0.31606239080429077, 0.285432904958725, 0.3901367783546448, 0.3063860535621643, 0.312701940536499, 0.3429080843925476, 0.36281368136405945, 0.2860463261604309, 0.3410678803920746, 0.30606693029403687, 0.3098071217536926, 0.3433891236782074, 0.337049663066864, 0.2844851016998291, 0.3203703761100769, 0.3284175992012024, 0.2743951082229614, 0.27600616216659546, 0.3081206679344177, 0.36312195658683777, 0.35184380412101746, 0.3211170434951782, 0.37896430492401123, 0.44468367099761963, 0.36475449800491333, 0.3426189124584198, 0.3603088855743408], "cos_apical_negg": [0.7844280004501343, 0.2979603707790375, 0.30254095792770386, 0.31606239080429077, 0.285432904958725, 0.3901367783546448, 0.3063860535621643, 0.312701940536499, 0.3429080843925476, 0.36281368136405945, 0.2860463261604309, 0.3410678803920746, 0.30606693029403687, 0.3098071217536926, 0.3433891236782074, 0.337049663066864, 0.2844851016998291, 0.3203703761100769, 0.3284175992012024, 0.2743951082229614, 0.27600616216659546, 0.3081206679344177, 0.36312195658683777, 0.35184380412101746, 0.3211170434951782, 0.37896430492401123, 0.44468367099761963, 0.36475449800491333, 0.3426189124584198, 0.3603088855743408], "cos_Ac_negg": [0.7844280004501343, 0.2979603707790375, 0.30254095792770386, 0.31606239080429077, 0.285432904958725, 0.3901367783546448, 0.3063860535621643, 0.312701940536499, 0.3429080843925476, 0.36281368136405945, 0.2860463261604309, 0.3410678803920746, 0.30606693029403687, 0.3098071217536926, 0.3433891236782074, 0.337049663066864, 0.2844851016998291, 0.3203703761100769, 0.3284175992012024, 0.2743951082229614, 0.27600616216659546, 0.3081206679344177, 0.36312195658683777, 0.35184380412101746, 0.3211170434951782, 0.37896430492401123, 0.44468367099761963, 0.36475449800491333, 0.3426189124584198, 0.3603088855743408]}}
\ No newline at end of file diff --git a/results/pilot_simvec_cifar10_sdil_w64_d60_s0.json b/results/pilot_simvec_cifar10_sdil_w64_d60_s0.json new file mode 100644 index 0000000..df99488 --- /dev/null +++ b/results/pilot_simvec_cifar10_sdil_w64_d60_s0.json @@ -0,0 +1 @@ +{"args": {"mode": "sdil", "dataset": "cifar10", "depth": 60, "width": 64, "act": "tanh", "residual": 1, "epochs": 5, "batch_size": 128, "eta": 0.02, "eta_A": 0.02, "eta_P": 0.002, "momentum": 0.9, "w_scale": 1.0, "a_scale": 1.0, "pert_sigma": 0.01, "pert_every": 4, "pert_ndirs": 16, "pert_mode": "simultaneous", "use_residual": 1, "learn_A": 1, "learn_P": 1, "p_neutral": 1, "p_warmup_steps": 200, "p_warmup_eta": 0.05, "nuis_rho": 0.0, "predictor_mode": "diagonal", "normalize_delta": 0, "settle_steps": 0, "kappa": 0.0, "feedback": "error", "seed": 0, "device": "cuda", "log_every": 500, "max_steps": 0, "probe_bs": 256, "outdir": "results", "tag": "pilot_simvec_cifar10_sdil_w64_d60_s0", "n_in": 3072}, "provenance": {"git_commit": "e0bfdfd51c78b945242bfa36a485d8773c4e17bb", "git_dirty": false}, "steps": [{"step": 0, "epoch": 0, "train_loss": 2.6154470443725586, "cos_r_negg": [0.00824391096830368, 0.03162052854895592, -0.047831855714321136, -0.0022021085023880005, -0.013032829388976097, -0.014932295307517052, -0.0704532340168953, -0.030040230602025986, -0.020447133108973503, -0.05151013284921646, 0.04950184002518654, -0.02089458890259266, 0.0030383728444576263, -0.05178888142108917, -0.05720045417547226, -0.06982016563415527, 0.08251547813415527, 0.04174470156431198, 0.05225787311792374, -0.03256798908114433, 0.023537758737802505, -0.008475148119032383, 0.07035811990499496, 0.018795045092701912, -0.008111847564578056, -0.011047604493796825, -0.06307624280452728, -0.004734567832201719, -0.0656220018863678, -0.0307687409222126, 0.0440501868724823, 0.004072677344083786, -0.02298356592655182, -0.032864805310964584, 0.0038536139763891697, 0.020486699417233467, -0.0055297925136983395, -0.0003991757985204458, 0.029928147792816162, 0.01716122217476368, 0.09282514452934265, 0.01995760202407837, -0.013349109329283237, 0.03773009404540062, -0.043992213904857635, 0.006984344683587551, -0.047033172100782394, -0.0005177173297852278, -0.05786652863025665, 0.04820503666996956, 0.005073794163763523, -0.10067319124937057, 0.030747149139642715, 0.029755573719739914, -0.01581287384033203, 0.042170196771621704, 0.013858259655535221, -0.06063992157578468, -0.048581261187791824, -0.031442247331142426], "cos_apical_negg": [0.00824391096830368, 0.03162052854895592, -0.047831855714321136, -0.0022021085023880005, -0.013032829388976097, -0.014932295307517052, -0.0704532340168953, -0.030040230602025986, -0.020447133108973503, -0.05151013284921646, 0.04950184002518654, -0.02089458890259266, 0.0030383728444576263, -0.05178888142108917, -0.05720045417547226, -0.06982016563415527, 0.08251547813415527, 0.04174470156431198, 0.05225787311792374, -0.03256798908114433, 0.023537758737802505, -0.008475148119032383, 0.07035811990499496, 0.018795045092701912, -0.008111847564578056, -0.011047604493796825, -0.06307624280452728, -0.004734567832201719, -0.0656220018863678, -0.0307687409222126, 0.0440501868724823, 0.004072677344083786, -0.02298356592655182, -0.032864805310964584, 0.0038536139763891697, 0.020486699417233467, -0.0055297925136983395, -0.0003991757985204458, 0.029928147792816162, 0.01716122217476368, 0.09282514452934265, 0.01995760202407837, -0.013349109329283237, 0.03773009404540062, -0.043992213904857635, 0.006984344683587551, -0.047033172100782394, -0.0005177173297852278, -0.05786652863025665, 0.04820503666996956, 0.005073794163763523, -0.10067319124937057, 0.030747149139642715, 0.029755573719739914, -0.01581287384033203, 0.042170196771621704, 0.013858259655535221, -0.06063992157578468, -0.048581261187791824, -0.031442247331142426], "cos_Ac_negg": [0.00824391096830368, 0.03162052854895592, -0.047831855714321136, -0.0022021085023880005, -0.013032829388976097, -0.014932295307517052, -0.0704532340168953, -0.030040230602025986, -0.020447133108973503, -0.05151013284921646, 0.04950184002518654, -0.02089458890259266, 0.0030383728444576263, -0.05178888142108917, -0.05720045417547226, -0.06982016563415527, 0.08251547813415527, 0.04174470156431198, 0.05225787311792374, -0.03256798908114433, 0.023537758737802505, -0.008475148119032383, 0.07035811990499496, 0.018795045092701912, -0.008111847564578056, -0.011047604493796825, -0.06307624280452728, -0.004734567832201719, -0.0656220018863678, -0.0307687409222126, 0.0440501868724823, 0.004072677344083786, -0.02298356592655182, -0.032864805310964584, 0.0038536139763891697, 0.020486699417233467, -0.0055297925136983395, -0.0003991757985204458, 0.029928147792816162, 0.01716122217476368, 0.09282514452934265, 0.01995760202407837, -0.013349109329283237, 0.03773009404540062, -0.043992213904857635, 0.006984344683587551, -0.047033172100782394, -0.0005177173297852278, -0.05786652863025665, 0.04820503666996956, 0.005073794163763523, -0.10067319124937057, 0.030747149139642715, 0.029755573719739914, -0.01581287384033203, 0.042170196771621704, 0.013858259655535221, -0.06063992157578468, -0.048581261187791824, -0.031442247331142426], "r_norm": [2.475647211074829, 2.582141876220703, 2.5103986263275146, 2.4314181804656982, 2.278536796569824, 2.4668307304382324, 2.378121852874756, 2.4344351291656494, 2.5928449630737305, 2.5062460899353027, 2.404479742050171, 2.582801103591919, 2.477358102798462, 2.4683475494384766, 2.603311538696289, 2.509106159210205, 2.5784785747528076, 2.5190746784210205, 2.4874401092529297, 2.52705717086792, 2.6225593090057373, 2.4242684841156006, 2.4262771606445312, 2.5080440044403076, 2.4646711349487305, 2.4284114837646484, 2.6363186836242676, 2.448397636413574, 2.3711695671081543, 2.640806198120117, 2.3413212299346924, 2.4465043544769287, 2.4522440433502197, 2.473102569580078, 2.573472023010254, 2.523059129714966, 2.4016785621643066, 2.382770538330078, 2.5407238006591797, 2.457386016845703, 2.526772975921631, 2.433110237121582, 2.478888511657715, 2.427353858947754, 2.5112290382385254, 2.5137362480163574, 2.5532827377319336, 2.4558825492858887, 2.407565116882324, 2.4302077293395996, 2.3953957557678223, 2.5483880043029785, 2.3778128623962402, 2.4375057220458984, 2.5560481548309326, 2.3974103927612305, 2.608649253845215, 2.4859652519226074, 2.3996095657348633, 2.4269697666168213], "ldr": {"dL_sdil": 0.014339923858642578, "dL_bp": -0.4264340400695801, "ratio": -0.03362753089857173}}, {"epoch_end": 0, "step": 391, "test_acc": 0.3654, "test_loss": 1.8139204467773438}, {"step": 500, "epoch": 1, "train_loss": 1.6714478731155396, "cos_r_negg": [0.6398711204528809, 0.08045336604118347, 0.06296725571155548, 0.03444591164588928, 0.04891926795244217, 0.09337839484214783, 0.06561367958784103, 0.07720770686864853, 0.06863469630479813, 0.02852531522512436, 0.0861290842294693, 0.021066123619675636, 0.06388528645038605, 0.033490002155303955, 0.06971554458141327, -0.002923880470916629, 0.038544632494449615, 0.06542747467756271, 0.1036529541015625, 0.0931016281247139, 0.024126440286636353, 0.10905695706605911, 0.12597110867500305, 0.09612717479467392, 0.12878364324569702, 0.032648757100105286, 0.07985586673021317, 0.01342830527573824, 0.017535505816340446, 0.0029379408806562424, 0.15186172723770142, 0.08372871577739716, 0.021085277199745178, 0.05216756463050842, 0.04087704420089722, 0.08477283269166946, 0.03431499004364014, 0.02604559063911438, 0.047012098133563995, 0.06130053848028183, 0.09004856646060944, 0.07965324074029922, 0.021382244303822517, 0.05597807094454765, 0.0441078245639801, 0.052947841584682465, 0.034906599670648575, 0.12976858019828796, -0.043593671172857285, 0.03311007469892502, -0.03262003883719444, 0.025063615292310715, 0.0022818464785814285, 0.09072045236825943, 0.17781250178813934, 0.1223243996500969, 0.11228570342063904, 0.1361311376094818, 0.002026174683123827, -0.003173322416841984], "cos_apical_negg": [0.6398711204528809, 0.08045336604118347, 0.06296725571155548, 0.03444591164588928, 0.04891926795244217, 0.09337839484214783, 0.06561367958784103, 0.07720770686864853, 0.06863469630479813, 0.02852531522512436, 0.0861290842294693, 0.021066123619675636, 0.06388528645038605, 0.033490002155303955, 0.06971554458141327, -0.002923880470916629, 0.038544632494449615, 0.06542747467756271, 0.1036529541015625, 0.0931016281247139, 0.024126440286636353, 0.10905695706605911, 0.12597110867500305, 0.09612717479467392, 0.12878364324569702, 0.032648757100105286, 0.07985586673021317, 0.01342830527573824, 0.017535505816340446, 0.0029379408806562424, 0.15186172723770142, 0.08372871577739716, 0.021085277199745178, 0.05216756463050842, 0.04087704420089722, 0.08477283269166946, 0.03431499004364014, 0.02604559063911438, 0.047012098133563995, 0.06130053848028183, 0.09004856646060944, 0.07965324074029922, 0.021382244303822517, 0.05597807094454765, 0.0441078245639801, 0.052947841584682465, 0.034906599670648575, 0.12976858019828796, -0.043593671172857285, 0.03311007469892502, -0.03262003883719444, 0.025063615292310715, 0.0022818464785814285, 0.09072045236825943, 0.17781250178813934, 0.1223243996500969, 0.11228570342063904, 0.1361311376094818, 0.002026174683123827, -0.003173322416841984], "cos_Ac_negg": [0.6398711204528809, 0.08045336604118347, 0.06296725571155548, 0.03444591164588928, 0.04891926795244217, 0.09337839484214783, 0.06561367958784103, 0.07720770686864853, 0.06863469630479813, 0.02852531522512436, 0.0861290842294693, 0.021066123619675636, 0.06388528645038605, 0.033490002155303955, 0.06971554458141327, -0.002923880470916629, 0.038544632494449615, 0.06542747467756271, 0.1036529541015625, 0.0931016281247139, 0.024126440286636353, 0.10905695706605911, 0.12597110867500305, 0.09612717479467392, 0.12878364324569702, 0.032648757100105286, 0.07985586673021317, 0.01342830527573824, 0.017535505816340446, 0.0029379408806562424, 0.15186172723770142, 0.08372871577739716, 0.021085277199745178, 0.05216756463050842, 0.04087704420089722, 0.08477283269166946, 0.03431499004364014, 0.02604559063911438, 0.047012098133563995, 0.06130053848028183, 0.09004856646060944, 0.07965324074029922, 0.021382244303822517, 0.05597807094454765, 0.0441078245639801, 0.052947841584682465, 0.034906599670648575, 0.12976858019828796, -0.043593671172857285, 0.03311007469892502, -0.03262003883719444, 0.025063615292310715, 0.0022818464785814285, 0.09072045236825943, 0.17781250178813934, 0.1223243996500969, 0.11228570342063904, 0.1361311376094818, 0.002026174683123827, -0.003173322416841984], "r_norm": [1.744222640991211, 1.7673091888427734, 1.719620704650879, 1.6692619323730469, 1.5701978206634521, 1.6794977188110352, 1.619675874710083, 1.6615722179412842, 1.7946276664733887, 1.7021353244781494, 1.6710517406463623, 1.7518370151519775, 1.6914194822311401, 1.6794383525848389, 1.779116153717041, 1.7089316844940186, 1.7661702632904053, 1.7161190509796143, 1.6989675760269165, 1.7464306354522705, 1.796947717666626, 1.6674470901489258, 1.6811712980270386, 1.716463565826416, 1.6950021982192993, 1.6596821546554565, 1.8044252395629883, 1.6744993925094604, 1.6209625005722046, 1.7967761754989624, 1.5931060314178467, 1.6828787326812744, 1.6644529104232788, 1.6855206489562988, 1.757262945175171, 1.7377161979675293, 1.6286098957061768, 1.6447557210922241, 1.7378298044204712, 1.6871581077575684, 1.7478718757629395, 1.6712108850479126, 1.6990809440612793, 1.676346778869629, 1.7159614562988281, 1.719051480293274, 1.742910623550415, 1.693921446800232, 1.6481950283050537, 1.6715093851089478, 1.640086054801941, 1.7570922374725342, 1.6490960121154785, 1.6771860122680664, 1.7548454999923706, 1.6541526317596436, 1.7746543884277344, 1.7031097412109375, 1.6500136852264404, 1.6536366939544678]}, {"epoch_end": 1, "step": 782, "test_acc": 0.3917, "test_loss": 1.7605324462890626}, {"step": 1000, "epoch": 2, "train_loss": 1.7386796474456787, "cos_r_negg": [0.7149659991264343, 0.1670122742652893, 0.09314488619565964, 0.11498928815126419, 0.12241406738758087, 0.12917405366897583, 0.11071436107158661, 0.1440773904323578, 0.1400883048772812, 0.1207822859287262, 0.14657053351402283, 0.07442326843738556, 0.14908245205879211, 0.06165171414613724, 0.13570082187652588, 0.03617776930332184, 0.07899796962738037, 0.14552965760231018, 0.15233905613422394, 0.18968290090560913, 0.09036578238010406, 0.15893997251987457, 0.1809094250202179, 0.15216019749641418, 0.198634535074234, 0.1756463348865509, 0.1495281159877777, 0.09190094470977783, 0.12810830771923065, 0.12022443860769272, 0.200030118227005, 0.17775483429431915, 0.07820558547973633, 0.11211375892162323, 0.12932491302490234, 0.20860761404037476, 0.1317947804927826, 0.08217453956604004, 0.09129787981510162, 0.12306429445743561, 0.16118690371513367, 0.1393539309501648, 0.09973818063735962, 0.13826069235801697, 0.09062434732913971, 0.11287806928157806, 0.14491289854049683, 0.18659758567810059, 0.05269630625844002, 0.061524514108896255, 0.09193314611911774, 0.10744161903858185, 0.04220355674624443, 0.16640610992908478, 0.22554731369018555, 0.15818336606025696, 0.11769498884677887, 0.16605177521705627, 0.0728539228439331, 0.08984527736902237], "cos_apical_negg": [0.7149659991264343, 0.1670122742652893, 0.09314488619565964, 0.11498928815126419, 0.12241406738758087, 0.12917405366897583, 0.11071436107158661, 0.1440773904323578, 0.1400883048772812, 0.1207822859287262, 0.14657053351402283, 0.07442326843738556, 0.14908245205879211, 0.06165171414613724, 0.13570082187652588, 0.03617776930332184, 0.07899796962738037, 0.14552965760231018, 0.15233905613422394, 0.18968290090560913, 0.09036578238010406, 0.15893997251987457, 0.1809094250202179, 0.15216019749641418, 0.198634535074234, 0.1756463348865509, 0.1495281159877777, 0.09190094470977783, 0.12810830771923065, 0.12022443860769272, 0.200030118227005, 0.17775483429431915, 0.07820558547973633, 0.11211375892162323, 0.12932491302490234, 0.20860761404037476, 0.1317947804927826, 0.08217453956604004, 0.09129787981510162, 0.12306429445743561, 0.16118690371513367, 0.1393539309501648, 0.09973818063735962, 0.13826069235801697, 0.09062434732913971, 0.11287806928157806, 0.14491289854049683, 0.18659758567810059, 0.05269630625844002, 0.061524514108896255, 0.09193314611911774, 0.10744161903858185, 0.04220355674624443, 0.16640610992908478, 0.22554731369018555, 0.15818336606025696, 0.11769498884677887, 0.16605177521705627, 0.0728539228439331, 0.08984527736902237], "cos_Ac_negg": [0.7149659991264343, 0.1670122742652893, 0.09314488619565964, 0.11498928815126419, 0.12241406738758087, 0.12917405366897583, 0.11071436107158661, 0.1440773904323578, 0.1400883048772812, 0.1207822859287262, 0.14657053351402283, 0.07442326843738556, 0.14908245205879211, 0.06165171414613724, 0.13570082187652588, 0.03617776930332184, 0.07899796962738037, 0.14552965760231018, 0.15233905613422394, 0.18968290090560913, 0.09036578238010406, 0.15893997251987457, 0.1809094250202179, 0.15216019749641418, 0.198634535074234, 0.1756463348865509, 0.1495281159877777, 0.09190094470977783, 0.12810830771923065, 0.12022443860769272, 0.200030118227005, 0.17775483429431915, 0.07820558547973633, 0.11211375892162323, 0.12932491302490234, 0.20860761404037476, 0.1317947804927826, 0.08217453956604004, 0.09129787981510162, 0.12306429445743561, 0.16118690371513367, 0.1393539309501648, 0.09973818063735962, 0.13826069235801697, 0.09062434732913971, 0.11287806928157806, 0.14491289854049683, 0.18659758567810059, 0.05269630625844002, 0.061524514108896255, 0.09193314611911774, 0.10744161903858185, 0.04220355674624443, 0.16640610992908478, 0.22554731369018555, 0.15818336606025696, 0.11769498884677887, 0.16605177521705627, 0.0728539228439331, 0.08984527736902237], "r_norm": [1.5149874687194824, 1.4545912742614746, 1.410078763961792, 1.3641705513000488, 1.2916669845581055, 1.3796054124832153, 1.3321301937103271, 1.3659336566925049, 1.4815492630004883, 1.3868237733840942, 1.3695979118347168, 1.4372308254241943, 1.3968360424041748, 1.3742424249649048, 1.4629924297332764, 1.392016053199768, 1.4400757551193237, 1.410008192062378, 1.397935390472412, 1.4373369216918945, 1.4703164100646973, 1.3764104843139648, 1.3879746198654175, 1.4060325622558594, 1.4013479948043823, 1.3597780466079712, 1.4914779663085938, 1.3664891719818115, 1.3288205862045288, 1.4693293571472168, 1.315602421760559, 1.3788049221038818, 1.3658448457717896, 1.3820130825042725, 1.4370286464691162, 1.4375009536743164, 1.3476827144622803, 1.3393909931182861, 1.4190698862075806, 1.3837302923202515, 1.4398962259292603, 1.3666224479675293, 1.3899409770965576, 1.375233769416809, 1.4023609161376953, 1.4056668281555176, 1.42929208278656, 1.3997068405151367, 1.3415281772613525, 1.3628301620483398, 1.3313851356506348, 1.430267095565796, 1.3402422666549683, 1.3733458518981934, 1.4488221406936646, 1.3627090454101562, 1.458683967590332, 1.402953863143921, 1.350010633468628, 1.3495091199874878]}, {"epoch_end": 2, "step": 1173, "test_acc": 0.4031, "test_loss": 1.7189087280273438}, {"step": 1500, "epoch": 3, "train_loss": 1.674216866493225, "cos_r_negg": [0.7562717795372009, 0.23670677840709686, 0.17480088770389557, 0.15152376890182495, 0.20917357504367828, 0.2414626032114029, 0.2050379365682602, 0.20528392493724823, 0.22965112328529358, 0.23302873969078064, 0.25278449058532715, 0.1806548833847046, 0.2373020350933075, 0.1885548084974289, 0.24202775955200195, 0.10442493855953217, 0.15905854105949402, 0.23355638980865479, 0.20943677425384521, 0.2593066692352295, 0.2014395296573639, 0.27389615774154663, 0.2723178565502167, 0.2842370867729187, 0.31301960349082947, 0.29388856887817383, 0.2478892207145691, 0.16036772727966309, 0.22407913208007812, 0.19607701897621155, 0.3110252618789673, 0.2773076593875885, 0.15167343616485596, 0.19562092423439026, 0.23808465898036957, 0.2769673466682434, 0.24957583844661713, 0.1914481818675995, 0.14318187534809113, 0.18917998671531677, 0.2828781306743622, 0.24087923765182495, 0.18703332543373108, 0.2174077033996582, 0.18807856738567352, 0.20496051013469696, 0.24633954465389252, 0.26210862398147583, 0.16335904598236084, 0.18114763498306274, 0.14670930802822113, 0.19647592306137085, 0.17429959774017334, 0.2453118860721588, 0.2778579890727997, 0.2434512823820114, 0.23584480583667755, 0.2681507468223572, 0.1965794414281845, 0.16891640424728394], "cos_apical_negg": [0.7562717795372009, 0.23670677840709686, 0.17480088770389557, 0.15152376890182495, 0.20917357504367828, 0.2414626032114029, 0.2050379365682602, 0.20528392493724823, 0.22965112328529358, 0.23302873969078064, 0.25278449058532715, 0.1806548833847046, 0.2373020350933075, 0.1885548084974289, 0.24202775955200195, 0.10442493855953217, 0.15905854105949402, 0.23355638980865479, 0.20943677425384521, 0.2593066692352295, 0.2014395296573639, 0.27389615774154663, 0.2723178565502167, 0.2842370867729187, 0.31301960349082947, 0.29388856887817383, 0.2478892207145691, 0.16036772727966309, 0.22407913208007812, 0.19607701897621155, 0.3110252618789673, 0.2773076593875885, 0.15167343616485596, 0.19562092423439026, 0.23808465898036957, 0.2769673466682434, 0.24957583844661713, 0.1914481818675995, 0.14318187534809113, 0.18917998671531677, 0.2828781306743622, 0.24087923765182495, 0.18703332543373108, 0.2174077033996582, 0.18807856738567352, 0.20496051013469696, 0.24633954465389252, 0.26210862398147583, 0.16335904598236084, 0.18114763498306274, 0.14670930802822113, 0.19647592306137085, 0.17429959774017334, 0.2453118860721588, 0.2778579890727997, 0.2434512823820114, 0.23584480583667755, 0.2681507468223572, 0.1965794414281845, 0.16891640424728394], "cos_Ac_negg": [0.7562717795372009, 0.23670677840709686, 0.17480088770389557, 0.15152376890182495, 0.20917357504367828, 0.2414626032114029, 0.2050379365682602, 0.20528392493724823, 0.22965112328529358, 0.23302873969078064, 0.25278449058532715, 0.1806548833847046, 0.2373020350933075, 0.1885548084974289, 0.24202775955200195, 0.10442493855953217, 0.15905854105949402, 0.23355638980865479, 0.20943677425384521, 0.2593066692352295, 0.2014395296573639, 0.27389615774154663, 0.2723178565502167, 0.2842370867729187, 0.31301960349082947, 0.29388856887817383, 0.2478892207145691, 0.16036772727966309, 0.22407913208007812, 0.19607701897621155, 0.3110252618789673, 0.2773076593875885, 0.15167343616485596, 0.19562092423439026, 0.23808465898036957, 0.2769673466682434, 0.24957583844661713, 0.1914481818675995, 0.14318187534809113, 0.18917998671531677, 0.2828781306743622, 0.24087923765182495, 0.18703332543373108, 0.2174077033996582, 0.18807856738567352, 0.20496051013469696, 0.24633954465389252, 0.26210862398147583, 0.16335904598236084, 0.18114763498306274, 0.14670930802822113, 0.19647592306137085, 0.17429959774017334, 0.2453118860721588, 0.2778579890727997, 0.2434512823820114, 0.23584480583667755, 0.2681507468223572, 0.1965794414281845, 0.16891640424728394], "r_norm": [1.344200611114502, 1.2108817100524902, 1.167618751525879, 1.1232117414474487, 1.0768123865127563, 1.1558520793914795, 1.120426058769226, 1.1290935277938843, 1.2262012958526611, 1.1577458381652832, 1.1429945230484009, 1.1914457082748413, 1.1572247743606567, 1.1355586051940918, 1.2218539714813232, 1.145140528678894, 1.1937854290008545, 1.1785221099853516, 1.1596720218658447, 1.1984870433807373, 1.2131530046463013, 1.149593710899353, 1.1569311618804932, 1.1768555641174316, 1.1786514520645142, 1.137153148651123, 1.2441513538360596, 1.1271854639053345, 1.1030635833740234, 1.220018744468689, 1.1088409423828125, 1.1552425622940063, 1.1316726207733154, 1.1554527282714844, 1.1870787143707275, 1.2072536945343018, 1.1363164186477661, 1.1089236736297607, 1.1688015460968018, 1.1612458229064941, 1.2006111145019531, 1.1386542320251465, 1.151556134223938, 1.1441164016723633, 1.1658554077148438, 1.1673924922943115, 1.1913270950317383, 1.1675288677215576, 1.1059856414794922, 1.1245050430297852, 1.0982189178466797, 1.1864988803863525, 1.1021993160247803, 1.1495025157928467, 1.2244726419448853, 1.1306880712509155, 1.2213971614837646, 1.1756497621536255, 1.1182398796081543, 1.1142864227294922]}, {"epoch_end": 3, "step": 1564, "test_acc": 0.4072, "test_loss": 1.7166262573242188}, {"epoch_end": 4, "step": 1955, "test_acc": 0.411, "test_loss": 1.7010625}], "final": {"test_acc": 0.411, "test_loss": 1.7010625, "wall_s": 90.28865337371826, "cos_r_negg": [0.7643067836761475, 0.3249530792236328, 0.2837162911891937, 0.2855551838874817, 0.30676549673080444, 0.3069310188293457, 0.30197370052337646, 0.30546045303344727, 0.32763174176216125, 0.3318811058998108, 0.3541974425315857, 0.28489792346954346, 0.34482961893081665, 0.2764572501182556, 0.30864521861076355, 0.2257709801197052, 0.26245182752609253, 0.35390183329582214, 0.33345910906791687, 0.36278486251831055, 0.302906334400177, 0.3676672577857971, 0.3594485819339752, 0.3552466928958893, 0.4007866382598877, 0.33722585439682007, 0.34977155923843384, 0.2813144326210022, 0.3393842577934265, 0.2692403793334961, 0.3742133677005768, 0.3639702796936035, 0.25902092456817627, 0.296712189912796, 0.3503630757331848, 0.3699854612350464, 0.33875665068626404, 0.3127887547016144, 0.26691317558288574, 0.3222803473472595, 0.38388729095458984, 0.37448370456695557, 0.30006590485572815, 0.31624943017959595, 0.2847444713115692, 0.336752712726593, 0.33294251561164856, 0.3756948411464691, 0.2714354395866394, 0.28742074966430664, 0.27885672450065613, 0.3194308280944824, 0.28793802857398987, 0.3472953736782074, 0.36703982949256897, 0.3526768684387207, 0.35906413197517395, 0.37517479062080383, 0.28327423334121704, 0.3131701946258545], "cos_apical_negg": [0.7643067836761475, 0.3249530792236328, 0.2837162911891937, 0.2855551838874817, 0.30676549673080444, 0.3069310188293457, 0.30197370052337646, 0.30546045303344727, 0.32763174176216125, 0.3318811058998108, 0.3541974425315857, 0.28489792346954346, 0.34482961893081665, 0.2764572501182556, 0.30864521861076355, 0.2257709801197052, 0.26245182752609253, 0.35390183329582214, 0.33345910906791687, 0.36278486251831055, 0.302906334400177, 0.3676672577857971, 0.3594485819339752, 0.3552466928958893, 0.4007866382598877, 0.33722585439682007, 0.34977155923843384, 0.2813144326210022, 0.3393842577934265, 0.2692403793334961, 0.3742133677005768, 0.3639702796936035, 0.25902092456817627, 0.296712189912796, 0.3503630757331848, 0.3699854612350464, 0.33875665068626404, 0.3127887547016144, 0.26691317558288574, 0.3222803473472595, 0.38388729095458984, 0.37448370456695557, 0.30006590485572815, 0.31624943017959595, 0.2847444713115692, 0.336752712726593, 0.33294251561164856, 0.3756948411464691, 0.2714354395866394, 0.28742074966430664, 0.27885672450065613, 0.3194308280944824, 0.28793802857398987, 0.3472953736782074, 0.36703982949256897, 0.3526768684387207, 0.35906413197517395, 0.37517479062080383, 0.28327423334121704, 0.3131701946258545], "cos_Ac_negg": [0.7643067836761475, 0.3249530792236328, 0.2837162911891937, 0.2855551838874817, 0.30676549673080444, 0.3069310188293457, 0.30197370052337646, 0.30546045303344727, 0.32763174176216125, 0.3318811058998108, 0.3541974425315857, 0.28489792346954346, 0.34482961893081665, 0.2764572501182556, 0.30864521861076355, 0.2257709801197052, 0.26245182752609253, 0.35390183329582214, 0.33345910906791687, 0.36278486251831055, 0.302906334400177, 0.3676672577857971, 0.3594485819339752, 0.3552466928958893, 0.4007866382598877, 0.33722585439682007, 0.34977155923843384, 0.2813144326210022, 0.3393842577934265, 0.2692403793334961, 0.3742133677005768, 0.3639702796936035, 0.25902092456817627, 0.296712189912796, 0.3503630757331848, 0.3699854612350464, 0.33875665068626404, 0.3127887547016144, 0.26691317558288574, 0.3222803473472595, 0.38388729095458984, 0.37448370456695557, 0.30006590485572815, 0.31624943017959595, 0.2847444713115692, 0.336752712726593, 0.33294251561164856, 0.3756948411464691, 0.2714354395866394, 0.28742074966430664, 0.27885672450065613, 0.3194308280944824, 0.28793802857398987, 0.3472953736782074, 0.36703982949256897, 0.3526768684387207, 0.35906413197517395, 0.37517479062080383, 0.28327423334121704, 0.3131701946258545]}}
\ No newline at end of file |