diff options
| -rw-r--r-- | RESIDUAL_MIRROR.md | 24 | ||||
| -rw-r--r-- | RESULTS.md | 25 | ||||
| -rw-r--r-- | REVIEW_SCORECARD.md | 2 | ||||
| -rw-r--r-- | ROADMAP.md | 9 | ||||
| -rw-r--r-- | results/residual_mirror_capture/fixed_hfa.json | 431 | ||||
| -rw-r--r-- | results/residual_mirror_capture/rrm_etaM0.03.json | 455 | ||||
| -rw-r--r-- | results/residual_mirror_capture/rrm_etaM0.1.json | 455 | ||||
| -rw-r--r-- | results/residual_mirror_capture/rrm_etaM0.3.json | 455 | ||||
| -rw-r--r-- | results/residual_mirror_capture_gate.json | 133 |
9 files changed, 1989 insertions, 0 deletions
diff --git a/RESIDUAL_MIRROR.md b/RESIDUAL_MIRROR.md index 312bc35..61e5691 100644 --- a/RESIDUAL_MIRROR.md +++ b/RESIDUAL_MIRROR.md @@ -71,3 +71,27 @@ same scalable substrate for a raw-versus-innovation mixed-traffic experiment. RRM is a baseline throughout. RRM-1/RRM-2 cannot raise the reviewer score; only a later load-bearing innovation result on top of it can do so. +## Audited RRM-1 outcome + +The frozen-forward capture gate passed on source revision `18aff5b`. The full +rate sweep is retained rather than reporting only the selected record: + +| method / rate | early alignment | all-layer alignment | feedback/forward cosine | norm range | +|:--|--:|--:|--:|:--| +| fixed HFA | -0.000263 | 0.011279 | 0.001614 | [0.959, 1.055] | +| RRM 0.03 | 0.075016 | 0.235001 | 0.622965 | [0.695, 0.792] | +| RRM 0.10 | **0.665070** | **0.730253** | **0.953680** | **[0.884, 1.018]** | +| RRM 0.30 | 0.003489 | 0.053744 | 0.740396 | [0.999, 30.132] | + +The preregistered selector chooses `eta_M=0.1`. All records have finite +reported metrics, and the selected record passes every alignment, cosine, +norm, and zero-query check. Its 20 observations cost `2.4458e9` MACs. The +high-rate instability remains visible and is not used to widen the rate grid. +RRM-2 is therefore opened at the frozen selected rate. + +Relative to normalized estimate-then-average mirroring at the same selected +rate and observation count, residual mirroring improves early alignment by +`0.218929`, all-layer alignment by `0.188094`, and feedback/forward cosine by +`0.038200`. This is evidence that removing estimator noise at the fixed point +matters. It remains inherited local predictive weight estimation, so the +strict reviewer score stays `5/10`. @@ -828,6 +828,31 @@ gain errors and the remaining compounding angular errors can materially change the trajectory. It would be misleading to present the capture result alone as BP-equivalent learning. +### Residual response-mirror capture + +Residual response mirroring changes the local baseline's update from averaging +a fresh noisy weight estimate to predicting each child response and updating +from the local response residual. At exact symmetry each stochastic update is +zero. The frozen RRM-1 screen gives: + +| method/rate | early alignment | all-layer alignment | feedback/forward cosine | norm range | +|:--|--:|--:|--:|:--| +| fixed HFA | -0.000263 | 0.011279 | 0.001614 | [0.959, 1.055] | +| RRM 0.03 | 0.075016 | 0.235001 | 0.622965 | [0.695, 0.792] | +| RRM 0.10 | **0.665070** | **0.730253** | **0.953680** | **[0.884, 1.018]** | +| RRM 0.30 | 0.003489 | 0.053744 | 0.740396 | [0.999, 30.132] | + +Selected `eta_M=0.1` passes every frozen capture check after 20 batch-1 local +observations. It consumes zero task-loss queries and `2.4458e9` MACs. Compared +with the selected normalized mirror record, it gains 0.219 early-alignment and +0.188 all-layer-alignment points while keeping feedback norms controlled. The +failed high-rate record shows that the residual rule is not rate-insensitive: +at 0.3 the maximum norm ratio reaches 30.13 and alignment collapses. + +RRM-2 is opened using the already frozen rate and cadence. This is a strong +inherited learned-alignment baseline, not Harnett-specific evidence, so it does +not raise the SDIL paper score by itself. + ## How to run `experiments/run.py --mode {bp,fa,dfa,sdil} --dataset {mnist,fmnist,cifar10} --depth D --residual {0,1} --act {tanh,gelu,silu,relu}` Batteries: `experiments/run_v2.sh <ds> "<depths>" <res> <act> "<seeds>" <ep> <pfx>`. diff --git a/REVIEW_SCORECARD.md b/REVIEW_SCORECARD.md index 93273ba..33ffb4e 100644 --- a/REVIEW_SCORECARD.md +++ b/REVIEW_SCORECARD.md @@ -64,6 +64,7 @@ Every formal result report records: | Hierarchical task-scalar V4 fails | 5 | Stable calibration leaves early alignment near zero; higher rates explode to 58.5x norm or become nonfinite after 800 queries | Correct mechanics and the right spatial family are insufficient when one global scalar estimates 267,904 feedback parameters | | Normalized response mirror capture passes | 5 | Twenty local observations reach 0.446 early alignment and 0.915 feedback/forward cosine at zero task-loss queries | Strongly resolves the engineering bottleneck, but all credit belongs to an inherited weight-estimation baseline until innovation is load-bearing | | Normalized response mirror short gate fails | 5 | WM reaches 64.04% and 0.939 early alignment at 0.9968x BP MACs, but misses its two accuracy gates by under one point | Strong inherited baseline and useful warning that alignment is insufficient; full run and any SDIL scale claim remain closed | +| Residual response mirror capture passes | 5 | RRM reaches 0.665 early and 0.730 all-layer alignment with 0.954 feedback/forward cosine after 20 zero-query local observations | Opens the frozen short accuracy gate, but remains inherited predictive weight estimation rather than SDIL evidence | | Oral-A A4 | not opened | The prerequisite A3 gate failed | No oral-A confirmation claim is available | These are conditional reviewer forecasts, not promised scores. A failed stage leaves its negative @@ -86,6 +87,7 @@ retroactively reopened by success on standard vision benchmarks. | 2026-07-22 / hierarchical V4-1 | Four frozen-forward records: etaA 0.1 leaves early alignment near zero, etaA 1 explodes one feedback norm to 58.5x, and etaA 10 is nonfinite | 5 → 5 | Closes global-task-scalar calibration of the full hierarchy at the fixed budget; richer local information is required before another accuracy endpoint | | 2026-07-22 / response-mirror WM-1 | Four clean frozen-forward records; selected etaM 0.1 reaches 0.4461 early and 0.5422 all-layer alignment with zero task-loss queries | 5 → 5 | Opens a strong inherited-baseline accuracy test and isolates information source as the bottleneck, but supplies no Harnett-specific task evidence | | 2026-07-22 / response-mirror WM-2 | Two clean short ResNet records; selected WM reaches 64.04% with 0.9393 early alignment and 0.9968x BP MACs, missing both accuracy gates narrowly | 5 → 5 | Substantially strengthens the comparator and cost story, but closes its full run and demonstrates that high credit cosine alone is not a scale result | +| 2026-07-22 / residual response-mirror RRM-1 | Four frozen-forward records; selected etaM 0.1 reaches 0.6651 early and 0.7303 all-layer alignment with controlled norms, zero task-loss queries, and 2.4458e9 MACs | 5 → 5 | Residual prediction removes much of the fixed-point estimator noise and opens the short task gate, but the gain belongs to an inherited baseline and does not establish somato-dendritic innovation | Future rows are appended only after an audited frozen stage. A score staying flat is informative: engineering, theory exposition, or visualization may make the paper more defensible without @@ -429,6 +429,15 @@ the within-10-point BP gate by 0.90 points, so WM-3 is closed. The next branch cannot tune this mirror recipe; it must make a substantive rule or information change and must not treat high alignment alone as task equivalence. +**Residual response-mirror RRM-1 status: capture gate passed.** Selected +etaM 0.1 reaches early/all alignment `0.665070/0.730253` and feedback/forward +cosine `0.953680` after 20 local observations, with norm ratios +`[0.884,1.018]`, zero task-loss queries, and `2.4458e9` MACs. All frozen gates +pass. EtaM 0.3 is visibly unstable (maximum norm ratio 30.13), so the selected +rate is not widened or retuned. The frozen two-rate RRM-2 short accuracy screen +is opened. This is inherited baseline progress and cannot change the paper +score until a load-bearing innovation experiment uses the substrate. + Prepare convolutional local-update primitives and ResNet-20/32/56 protocols early. Queue frozen runs opportunistically on authorized idle GPUs. Because BurstCCN already reports CIFAR-10 and ImageNet scaling, dataset scale alone is not novel. The oral-level target is a memorable joint diff --git a/results/residual_mirror_capture/fixed_hfa.json b/results/residual_mirror_capture/fixed_hfa.json new file mode 100644 index 0000000..15a76cb --- /dev/null +++ b/results/residual_mirror_capture/fixed_hfa.json @@ -0,0 +1,431 @@ +{ + "architecture": { + "adaptive_apical_parameters": 0, + "adaptive_feedback_parameters": 0, + "base_width": 16, + "blocks_per_stage": 3, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "depth": 20, + "family": "CIFAR 6n+2 ResNet, option-A shortcuts", + "fixed_feedback_parameters": 267904, + "fixed_traffic_coefficients": 0, + "forward_parameters": 269722, + "hidden_shapes": [ + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ] + ], + "normalization": "batchnorm", + "predictor_parameters": 0, + "residual_scale": 1.0, + "vectorizer_mode": null, + "vectorizer_parameters": 0 + }, + "args": { + "a_scale": 1.0, + "a_warmup_steps": 0, + "alignment_probe": 64, + "apical_calibration_mode": "unit_targets", + "apical_seed": null, + "augment_train": 1, + "batch_size": 128, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "data_dir": "/home/yurenh2/sdrn/data", + "depth": 20, + "device": "cuda", + "epochs": 0, + "eta_A": 0.01, + "eta_P": 0.01, + "eval_every": 0, + "eval_split": "validation", + "learn_P": 0, + "loader_seed": 0, + "lr": 0.1, + "lr_gamma": 0.1, + "lr_milestones": "100,150", + "lr_schedule": "constant", + "max_steps": 0, + "mirror_batch_size": 1, + "mirror_eta": 0.1, + "mirror_every": 16, + "mirror_noise_std": 1.0, + "mirror_seed": 3000, + "mirror_warmup_steps": 0, + "mode": "hfa", + "momentum": 0.9, + "normalization": "batchnorm", + "nuisance_scale": 0.0, + "out": "results/residual_mirror_capture/fixed_hfa.json", + "output_lr": 0.1, + "pert_directions": 1, + "pert_every": 4, + "pert_sigma": 0.01, + "perturb_seed": 1000, + "predictor_warmup_steps": 0, + "residual_scale": null, + "seed": 0, + "split_seed": 2027, + "train_limit": 10000, + "use_residual": 1, + "val_examples": 5000, + "vectorizer_mode": "spatial_template", + "warmup_epochs": 0, + "weight_decay": 0.0001, + "weight_scale": 1.0, + "width": 16 + }, + "calibration_metric_space": null, + "counters": { + "apical_warmup_examples": 0, + "calibration_event_examples": 0, + "causal_scalar_observations": 0, + "logical_batch_loss_queries": 0, + "mirror_conv_examples": 0, + "mirror_events": 0, + "mirror_readout_examples": 0, + "ordinary_examples": 0, + "per_example_loss_terms": 0, + "perturbation_events": 0, + "perturbation_forward_examples": 0, + "predictor_warmup_examples": 0 + }, + "diagnostics": { + "early_third_mean": -0.00026315699263553444, + "feedback_forward_cosine": [ + 0.0008809715509414673, + -0.002320552244782448, + 0.03730485588312149, + 0.02497659996151924, + -0.03438456356525421, + -0.001257734838873148, + 0.023045765236020088, + 0.010684739798307419, + -0.013966654427349567, + -0.03158111870288849, + 0.009583890438079834, + -9.433389641344547e-05, + 0.004490766208618879, + -0.004784357734024525, + 0.004104882013052702, + 0.004543168004602194, + -0.005818712059408426, + 0.0014950999757274985, + 0.003767170011997223 + ], + "feedback_forward_norm_ratio": [ + 0.9996462464332581, + 1.0550107955932617, + 0.9593725800514221, + 1.0017613172531128, + 0.999921441078186, + 0.9631232619285583, + 0.9992203712463379, + 0.9976226687431335, + 1.0018895864486694, + 1.0030434131622314, + 1.0180330276489258, + 1.0107296705245972, + 0.9962885975837708, + 0.9891077876091003, + 1.0009011030197144, + 1.0021207332611084, + 1.0043002367019653, + 1.0021206140518188, + 0.9992877244949341 + ], + "innovation_negative_gradient_cosine": [ + -0.0005718545289710164, + -0.0006265087286010385, + 0.0012021246366202831, + 3.765033034142107e-05, + 0.00048483326099812984, + -0.002105186926200986, + 0.003292462322860956, + 0.0002729812986217439, + 0.007543377578258514, + -0.005317358300089836, + 0.009830433875322342, + -0.001097709871828556, + 0.023688670247793198, + 0.007433340419083834, + 0.04359162598848343, + 0.0057456037029623985, + 0.039965368807315826, + 0.0030560996383428574, + 0.07787258923053741 + ], + "normalization_state": "training_batch_stats_without_running_update", + "raw_negative_gradient_cosine": [ + -0.0005718545289710164, + -0.0006265087286010385, + 0.0012021246366202831, + 3.765033034142107e-05, + 0.00048483326099812984, + -0.002105186926200986, + 0.003292462322860956, + 0.0002729812986217439, + 0.007543377578258514, + -0.005317358300089836, + 0.009830433875322342, + -0.001097709871828556, + 0.023688670247793198, + 0.007433340419083834, + 0.04359162598848343, + 0.0057456037029623985, + 0.039965368807315826, + 0.0030560996383428574, + 0.07787258923053741 + ], + "teaching_negative_gradient_cosine": [ + -0.0005718545289710164, + -0.0006265087286010385, + 0.0012021246366202831, + 3.765033034142107e-05, + 0.00048483326099812984, + -0.002105186926200986, + 0.003292462322860956, + 0.0002729812986217439, + 0.007543377578258514, + -0.005317358300089836, + 0.009830433875322342, + -0.001097709871828556, + 0.023688670247793198, + 0.007433340419083834, + 0.04359162598848343, + 0.0057456037029623985, + 0.039965368807315826, + 0.0030560996383428574, + 0.07787258923053741 + ], + "wall_s": 0.20391488075256348 + }, + "epochs": [], + "evaluation_protocol": { + "test_evaluations": 0, + "test_used_for_selection": false, + "validation_evaluations": 1 + }, + "final": { + "accuracy": 0.1004, + "epoch": 0, + "evaluation_split": "validation", + "finite": true, + "loss": 16.593759375, + "step": 0 + }, + "hardware": { + "cuda_device_name": "NVIDIA GeForce GTX 1080", + "cuda_visible_devices": "5", + "device": "cuda", + "device_total_memory_bytes": 8507949056, + "peak_memory_allocated_bytes": 1730765824, + "peak_memory_reserved_bytes": 2168455168, + "torch_version": "2.3.1+cu118" + }, + "protocol_family": "oral_a_cifar_local_resnet_development", + "provenance": { + "git_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "git_tracked_dirty": false + }, + "schema_version": 1, + "split": { + "cifar_source_files": [ + { + "bytes": 31035704, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_1", + "sha256": "54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd" + }, + { + "bytes": 31035320, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_2", + "sha256": "766b2cef9fbc745cf056b3152224f7cf77163b330ea9a15f9392beb8b89bc5a8" + }, + { + "bytes": 31035999, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_3", + "sha256": "0f00d98ebfb30b3ec0ad19f9756dc2630b89003e10525f5e148445e82aa6a1f9" + }, + { + "bytes": 31035696, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_4", + "sha256": "3f7bb240661948b8f4d53e36ec720d8306f5668bd0071dcb4e6c947f78e9682b" + }, + { + "bytes": 31035623, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_5", + "sha256": "d91802434d8376bbaeeadf58a737e3a1b12ac839077e931237e0dcd43adcb154" + }, + { + "bytes": 31035526, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/test_batch", + "sha256": "f53d8d457504f7cff4ea9e021afcf0e0ad8e24a91f3fc42091b8adef61157831" + } + ], + "dataset": "cifar10", + "evaluation_split": "validation", + "input_layout": "NCHW", + "input_shape": [ + 3, + 32, + 32 + ], + "loader_seed": 0, + "normalization_mean": [ + 0.49140000343322754, + 0.4821999967098236, + 0.4465000033378601 + ], + "normalization_std": [ + 0.24699999392032623, + 0.2434999942779541, + 0.26159998774528503 + ], + "split_from_training_only": true, + "split_seed": 2027, + "test_examples": 10000, + "train_examples": 10000, + "training_augmentation": "random_crop_32_padding_4_zero_then_horizontal_flip_p0.5", + "validation_class_counts": { + "0": 500, + "1": 500, + "2": 500, + "3": 500, + "4": 500, + "5": 500, + "6": 500, + "7": 500, + "8": 500, + "9": 500 + }, + "validation_examples": 5000, + "validation_index_sha256": "8328b206a97c420e49e54e3eca4abe3274c4756b084355784ea3fb8059e4515b" + }, + "timing": { + "apical_warmup_wall_s": 0.0, + "evaluation_wall_s": 0.6160025596618652, + "mirror_warmup_wall_s": 0.0, + "predictor_warmup_wall_s": 0.0, + "timing_excludes_data_loading_hashing_and_model_construction": true, + "total_timed_wall_s": 0.6161468029022217, + "train_wall_s": 0.0 + }, + "work": { + "apical_macs_per_example": 40108672, + "causal_scalar_observations": 0, + "components": { + "apical_projection_macs": 0, + "apical_regression_macs": 0, + "bp_reverse_macs_estimate": 0, + "local_weight_correlation_macs": 0, + "mirror_feedback_prediction_macs": 0, + "mirror_local_correlation_macs": 0, + "mirror_response_macs": 0, + "ordinary_forward_macs": 0, + "perturbation_forward_macs": 0, + "warmup_clean_forward_macs": 0 + }, + "definition": "multiply-accumulates in conv/linear maps; one local weight correlation equals one forward-weight MAC count; BP reverse is estimated as one weight-gradient plus one activation-gradient convolution per forward convolution; elementwise nonlinearities and optimizer arithmetic excluded", + "forward_macs_per_example": 40551040, + "logical_batch_loss_queries": 0, + "mirror_probe_examples": 0, + "mirror_readout_probe_examples": 0, + "per_example_cross_entropy_terms": 0, + "total_clean_forward_examples": 0, + "total_forward_equivalent_examples": 0, + "total_macs_estimate": 0 + } +} diff --git a/results/residual_mirror_capture/rrm_etaM0.03.json b/results/residual_mirror_capture/rrm_etaM0.03.json new file mode 100644 index 0000000..e94b008 --- /dev/null +++ b/results/residual_mirror_capture/rrm_etaM0.03.json @@ -0,0 +1,455 @@ +{ + "architecture": { + "adaptive_apical_parameters": 0, + "adaptive_feedback_parameters": 267904, + "base_width": 16, + "blocks_per_stage": 3, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "depth": 20, + "family": "CIFAR 6n+2 ResNet, option-A shortcuts", + "fixed_feedback_parameters": 0, + "fixed_traffic_coefficients": 0, + "forward_parameters": 269722, + "hidden_shapes": [ + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ] + ], + "normalization": "batchnorm", + "predictor_parameters": 0, + "residual_scale": 1.0, + "vectorizer_mode": null, + "vectorizer_parameters": 0 + }, + "args": { + "a_scale": 1.0, + "a_warmup_steps": 0, + "alignment_probe": 64, + "apical_calibration_mode": "unit_targets", + "apical_seed": null, + "augment_train": 1, + "batch_size": 128, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "data_dir": "/home/yurenh2/sdrn/data", + "depth": 20, + "device": "cuda", + "epochs": 0, + "eta_A": 0.01, + "eta_P": 0.01, + "eval_every": 0, + "eval_split": "validation", + "learn_P": 0, + "loader_seed": 0, + "lr": 0.1, + "lr_gamma": 0.1, + "lr_milestones": "100,150", + "lr_schedule": "constant", + "max_steps": 0, + "mirror_batch_size": 1, + "mirror_eta": 0.03, + "mirror_every": 16, + "mirror_noise_std": 1.0, + "mirror_seed": 3000, + "mirror_warmup_steps": 20, + "mode": "rrm", + "momentum": 0.9, + "normalization": "batchnorm", + "nuisance_scale": 0.0, + "out": "results/residual_mirror_capture/rrm_etaM0.03.json", + "output_lr": 0.1, + "pert_directions": 1, + "pert_every": 4, + "pert_sigma": 0.01, + "perturb_seed": 1000, + "predictor_warmup_steps": 0, + "residual_scale": null, + "seed": 0, + "split_seed": 2027, + "train_limit": 10000, + "use_residual": 1, + "val_examples": 5000, + "vectorizer_mode": "spatial_template", + "warmup_epochs": 0, + "weight_decay": 0.0001, + "weight_scale": 1.0, + "width": 16 + }, + "calibration_metric_space": "local_parent_child_response_residual", + "counters": { + "apical_warmup_examples": 0, + "calibration_event_examples": 0, + "causal_scalar_observations": 0, + "logical_batch_loss_queries": 0, + "mirror_conv_examples": 20, + "mirror_events": 20, + "mirror_readout_examples": 20480, + "ordinary_examples": 0, + "per_example_loss_terms": 0, + "perturbation_events": 0, + "perturbation_forward_examples": 0, + "predictor_warmup_examples": 0 + }, + "diagnostics": { + "early_third_mean": 0.07501635327935219, + "feedback_forward_cosine": [ + 0.642231822013855, + 0.6209580898284912, + 0.6709350347518921, + 0.6560195088386536, + 0.6285985112190247, + 0.6578096747398376, + 0.6539484262466431, + 0.6415038704872131, + 0.625426709651947, + 0.6155803203582764, + 0.6325298547744751, + 0.6294562816619873, + 0.609635591506958, + 0.5829833745956421, + 0.5823480486869812, + 0.5837353467941284, + 0.573731005191803, + 0.584334135055542, + 0.6445740461349487 + ], + "feedback_forward_norm_ratio": [ + 0.7110515832901001, + 0.7323321104049683, + 0.7069358825683594, + 0.7193056344985962, + 0.6983659863471985, + 0.6949356198310852, + 0.7238355875015259, + 0.723031222820282, + 0.7164431214332581, + 0.710602343082428, + 0.7304889559745789, + 0.7230876684188843, + 0.7481958866119385, + 0.7847121357917786, + 0.7924403548240662, + 0.7900992631912231, + 0.7892081141471863, + 0.787951648235321, + 0.711616039276123 + ], + "innovation_negative_gradient_cosine": [ + 0.042576856911182404, + 0.04807133227586746, + 0.07582877576351166, + 0.07072202116250992, + 0.10915401577949524, + 0.10374511778354645, + 0.14519003033638, + 0.12541726231575012, + 0.19553722441196442, + 0.16349542140960693, + 0.25410711765289307, + 0.2118675708770752, + 0.310130774974823, + 0.24106627702713013, + 0.4149003028869629, + 0.32370972633361816, + 0.5440131425857544, + 0.39865854382514954, + 0.6868242621421814 + ], + "normalization_state": "training_batch_stats_without_running_update", + "raw_negative_gradient_cosine": [ + 0.042576856911182404, + 0.04807133227586746, + 0.07582877576351166, + 0.07072202116250992, + 0.10915401577949524, + 0.10374511778354645, + 0.14519003033638, + 0.12541726231575012, + 0.19553722441196442, + 0.16349542140960693, + 0.25410711765289307, + 0.2118675708770752, + 0.310130774974823, + 0.24106627702713013, + 0.4149003028869629, + 0.32370972633361816, + 0.5440131425857544, + 0.39865854382514954, + 0.6868242621421814 + ], + "teaching_negative_gradient_cosine": [ + 0.042576856911182404, + 0.04807133227586746, + 0.07582877576351166, + 0.07072202116250992, + 0.10915401577949524, + 0.10374511778354645, + 0.14519003033638, + 0.12541726231575012, + 0.19553722441196442, + 0.16349542140960693, + 0.25410711765289307, + 0.2118675708770752, + 0.310130774974823, + 0.24106627702713013, + 0.4149003028869629, + 0.32370972633361816, + 0.5440131425857544, + 0.39865854382514954, + 0.6868242621421814 + ], + "wall_s": 0.1743333339691162 + }, + "epochs": [], + "evaluation_protocol": { + "test_evaluations": 0, + "test_used_for_selection": false, + "validation_evaluations": 1 + }, + "final": { + "accuracy": 0.1004, + "epoch": 0, + "evaluation_split": "validation", + "finite": true, + "loss": 16.593759375, + "step": 0 + }, + "hardware": { + "cuda_device_name": "NVIDIA GeForce GTX 1080", + "cuda_visible_devices": "7", + "device": "cuda", + "device_total_memory_bytes": 8507949056, + "peak_memory_allocated_bytes": 1730634752, + "peak_memory_reserved_bytes": 2181038080, + "torch_version": "2.3.1+cu118" + }, + "mirror_warmup": { + "first": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 1.4124661316345333, + "mirror_response_residual_rms": 1.90927576279183, + "mirror_update_rms": 0.25520769706611623, + "readout_batch_size": 1024 + }, + "last": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 0.8051068529504194, + "mirror_response_residual_rms": 1.090256044612914, + "mirror_update_rms": 0.15445659525963165, + "readout_batch_size": 1024 + }, + "mean": { + "conv_batch_size": 1.0, + "mirror_response_residual_fraction": 1.0790779544942406, + "mirror_response_residual_rms": 1.461456808634224, + "mirror_update_rms": 0.19879612560335747, + "readout_batch_size": 1024.0 + }, + "steps": 20 + }, + "protocol_family": "oral_a_cifar_local_resnet_development", + "provenance": { + "git_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "git_tracked_dirty": false + }, + "schema_version": 1, + "split": { + "cifar_source_files": [ + { + "bytes": 31035704, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_1", + "sha256": "54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd" + }, + { + "bytes": 31035320, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_2", + "sha256": "766b2cef9fbc745cf056b3152224f7cf77163b330ea9a15f9392beb8b89bc5a8" + }, + { + "bytes": 31035999, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_3", + "sha256": "0f00d98ebfb30b3ec0ad19f9756dc2630b89003e10525f5e148445e82aa6a1f9" + }, + { + "bytes": 31035696, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_4", + "sha256": "3f7bb240661948b8f4d53e36ec720d8306f5668bd0071dcb4e6c947f78e9682b" + }, + { + "bytes": 31035623, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_5", + "sha256": "d91802434d8376bbaeeadf58a737e3a1b12ac839077e931237e0dcd43adcb154" + }, + { + "bytes": 31035526, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/test_batch", + "sha256": "f53d8d457504f7cff4ea9e021afcf0e0ad8e24a91f3fc42091b8adef61157831" + } + ], + "dataset": "cifar10", + "evaluation_split": "validation", + "input_layout": "NCHW", + "input_shape": [ + 3, + 32, + 32 + ], + "loader_seed": 0, + "normalization_mean": [ + 0.49140000343322754, + 0.4821999967098236, + 0.4465000033378601 + ], + "normalization_std": [ + 0.24699999392032623, + 0.2434999942779541, + 0.26159998774528503 + ], + "split_from_training_only": true, + "split_seed": 2027, + "test_examples": 10000, + "train_examples": 10000, + "training_augmentation": "random_crop_32_padding_4_zero_then_horizontal_flip_p0.5", + "validation_class_counts": { + "0": 500, + "1": 500, + "2": 500, + "3": 500, + "4": 500, + "5": 500, + "6": 500, + "7": 500, + "8": 500, + "9": 500 + }, + "validation_examples": 5000, + "validation_index_sha256": "8328b206a97c420e49e54e3eca4abe3274c4756b084355784ea3fb8059e4515b" + }, + "timing": { + "apical_warmup_wall_s": 0.0, + "evaluation_wall_s": 0.442340612411499, + "mirror_warmup_wall_s": 0.5587825775146484, + "predictor_warmup_wall_s": 0.0, + "timing_excludes_data_loading_hashing_and_model_construction": true, + "total_timed_wall_s": 1.0013489723205566, + "train_wall_s": 0.0 + }, + "work": { + "apical_macs_per_example": 40108672, + "causal_scalar_observations": 0, + "components": { + "apical_projection_macs": 0, + "apical_regression_macs": 0, + "bp_reverse_macs_estimate": 0, + "local_weight_correlation_macs": 0, + "mirror_feedback_prediction_macs": 815267840, + "mirror_local_correlation_macs": 815267840, + "mirror_response_macs": 815267840, + "ordinary_forward_macs": 0, + "perturbation_forward_macs": 0, + "warmup_clean_forward_macs": 0 + }, + "definition": "multiply-accumulates in conv/linear maps; one local weight correlation equals one forward-weight MAC count; BP reverse is estimated as one weight-gradient plus one activation-gradient convolution per forward convolution; elementwise nonlinearities and optimizer arithmetic excluded", + "forward_macs_per_example": 40551040, + "logical_batch_loss_queries": 0, + "mirror_probe_examples": 20, + "mirror_readout_probe_examples": 20480, + "per_example_cross_entropy_terms": 0, + "total_clean_forward_examples": 0, + "total_forward_equivalent_examples": 0, + "total_macs_estimate": 2445803520 + } +} diff --git a/results/residual_mirror_capture/rrm_etaM0.1.json b/results/residual_mirror_capture/rrm_etaM0.1.json new file mode 100644 index 0000000..eeaad1e --- /dev/null +++ b/results/residual_mirror_capture/rrm_etaM0.1.json @@ -0,0 +1,455 @@ +{ + "architecture": { + "adaptive_apical_parameters": 0, + "adaptive_feedback_parameters": 267904, + "base_width": 16, + "blocks_per_stage": 3, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "depth": 20, + "family": "CIFAR 6n+2 ResNet, option-A shortcuts", + "fixed_feedback_parameters": 0, + "fixed_traffic_coefficients": 0, + "forward_parameters": 269722, + "hidden_shapes": [ + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ] + ], + "normalization": "batchnorm", + "predictor_parameters": 0, + "residual_scale": 1.0, + "vectorizer_mode": null, + "vectorizer_parameters": 0 + }, + "args": { + "a_scale": 1.0, + "a_warmup_steps": 0, + "alignment_probe": 64, + "apical_calibration_mode": "unit_targets", + "apical_seed": null, + "augment_train": 1, + "batch_size": 128, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "data_dir": "/home/yurenh2/sdrn/data", + "depth": 20, + "device": "cuda", + "epochs": 0, + "eta_A": 0.01, + "eta_P": 0.01, + "eval_every": 0, + "eval_split": "validation", + "learn_P": 0, + "loader_seed": 0, + "lr": 0.1, + "lr_gamma": 0.1, + "lr_milestones": "100,150", + "lr_schedule": "constant", + "max_steps": 0, + "mirror_batch_size": 1, + "mirror_eta": 0.1, + "mirror_every": 16, + "mirror_noise_std": 1.0, + "mirror_seed": 3000, + "mirror_warmup_steps": 20, + "mode": "rrm", + "momentum": 0.9, + "normalization": "batchnorm", + "nuisance_scale": 0.0, + "out": "results/residual_mirror_capture/rrm_etaM0.1.json", + "output_lr": 0.1, + "pert_directions": 1, + "pert_every": 4, + "pert_sigma": 0.01, + "perturb_seed": 1000, + "predictor_warmup_steps": 0, + "residual_scale": null, + "seed": 0, + "split_seed": 2027, + "train_limit": 10000, + "use_residual": 1, + "val_examples": 5000, + "vectorizer_mode": "spatial_template", + "warmup_epochs": 0, + "weight_decay": 0.0001, + "weight_scale": 1.0, + "width": 16 + }, + "calibration_metric_space": "local_parent_child_response_residual", + "counters": { + "apical_warmup_examples": 0, + "calibration_event_examples": 0, + "causal_scalar_observations": 0, + "logical_batch_loss_queries": 0, + "mirror_conv_examples": 20, + "mirror_events": 20, + "mirror_readout_examples": 20480, + "ordinary_examples": 0, + "per_example_loss_terms": 0, + "perturbation_events": 0, + "perturbation_forward_examples": 0, + "predictor_warmup_examples": 0 + }, + "diagnostics": { + "early_third_mean": 0.6650698681672415, + "feedback_forward_cosine": [ + 0.9899365305900574, + 0.9889124631881714, + 0.9906322360038757, + 0.9903721213340759, + 0.9900292158126831, + 0.9908440113067627, + 0.9880079030990601, + 0.9845560789108276, + 0.9840002059936523, + 0.9835241436958313, + 0.9841198921203613, + 0.984214723110199, + 0.9532983303070068, + 0.8680597543716431, + 0.8650628924369812, + 0.8668069839477539, + 0.8626937866210938, + 0.864578366279602, + 0.9902748465538025 + ], + "feedback_forward_norm_ratio": [ + 0.8874651789665222, + 0.8877098560333252, + 0.8901256322860718, + 0.8916847109794617, + 0.8841801881790161, + 0.8877401351928711, + 0.8954755663871765, + 0.8946005702018738, + 0.8906601071357727, + 0.8881114721298218, + 0.8941957354545593, + 0.8923366665840149, + 0.9199835062026978, + 1.0136339664459229, + 1.0178089141845703, + 1.0176160335540771, + 1.0177607536315918, + 1.0179650783538818, + 0.8878191709518433 + ], + "innovation_negative_gradient_cosine": [ + 0.6502600908279419, + 0.6562603712081909, + 0.6643938422203064, + 0.6675176620483398, + 0.6731520891189575, + 0.6788351535797119, + 0.6819244027137756, + 0.6839392781257629, + 0.6955236196517944, + 0.699924886226654, + 0.7122894525527954, + 0.7139394283294678, + 0.7200665473937988, + 0.6781741380691528, + 0.7831467390060425, + 0.7739078998565674, + 0.8912925720214844, + 0.8592211604118347, + 0.991040825843811 + ], + "normalization_state": "training_batch_stats_without_running_update", + "raw_negative_gradient_cosine": [ + 0.6502600908279419, + 0.6562603712081909, + 0.6643938422203064, + 0.6675176620483398, + 0.6731520891189575, + 0.6788351535797119, + 0.6819244027137756, + 0.6839392781257629, + 0.6955236196517944, + 0.699924886226654, + 0.7122894525527954, + 0.7139394283294678, + 0.7200665473937988, + 0.6781741380691528, + 0.7831467390060425, + 0.7739078998565674, + 0.8912925720214844, + 0.8592211604118347, + 0.991040825843811 + ], + "teaching_negative_gradient_cosine": [ + 0.6502600908279419, + 0.6562603712081909, + 0.6643938422203064, + 0.6675176620483398, + 0.6731520891189575, + 0.6788351535797119, + 0.6819244027137756, + 0.6839392781257629, + 0.6955236196517944, + 0.699924886226654, + 0.7122894525527954, + 0.7139394283294678, + 0.7200665473937988, + 0.6781741380691528, + 0.7831467390060425, + 0.7739078998565674, + 0.8912925720214844, + 0.8592211604118347, + 0.991040825843811 + ], + "wall_s": 0.15188956260681152 + }, + "epochs": [], + "evaluation_protocol": { + "test_evaluations": 0, + "test_used_for_selection": false, + "validation_evaluations": 1 + }, + "final": { + "accuracy": 0.1004, + "epoch": 0, + "evaluation_split": "validation", + "finite": true, + "loss": 16.593759375, + "step": 0 + }, + "hardware": { + "cuda_device_name": "NVIDIA GeForce GTX 1080", + "cuda_visible_devices": "5", + "device": "cuda", + "device_total_memory_bytes": 8507949056, + "peak_memory_allocated_bytes": 1730634752, + "peak_memory_reserved_bytes": 2181038080, + "torch_version": "2.3.1+cu118" + }, + "mirror_warmup": { + "first": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 1.4124661316345333, + "mirror_response_residual_rms": 1.90927576279183, + "mirror_update_rms": 0.25520769706611623, + "readout_batch_size": 1024 + }, + "last": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 0.26271250078043745, + "mirror_response_residual_rms": 0.3557588547676734, + "mirror_update_rms": 0.0907495165488641, + "readout_batch_size": 1024 + }, + "mean": { + "conv_batch_size": 1.0, + "mirror_response_residual_fraction": 0.6719071790962137, + "mirror_response_residual_rms": 0.9100978838204868, + "mirror_update_rms": 0.15685383656211757, + "readout_batch_size": 1024.0 + }, + "steps": 20 + }, + "protocol_family": "oral_a_cifar_local_resnet_development", + "provenance": { + "git_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "git_tracked_dirty": false + }, + "schema_version": 1, + "split": { + "cifar_source_files": [ + { + "bytes": 31035704, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_1", + "sha256": "54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd" + }, + { + "bytes": 31035320, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_2", + "sha256": "766b2cef9fbc745cf056b3152224f7cf77163b330ea9a15f9392beb8b89bc5a8" + }, + { + "bytes": 31035999, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_3", + "sha256": "0f00d98ebfb30b3ec0ad19f9756dc2630b89003e10525f5e148445e82aa6a1f9" + }, + { + "bytes": 31035696, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_4", + "sha256": "3f7bb240661948b8f4d53e36ec720d8306f5668bd0071dcb4e6c947f78e9682b" + }, + { + "bytes": 31035623, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_5", + "sha256": "d91802434d8376bbaeeadf58a737e3a1b12ac839077e931237e0dcd43adcb154" + }, + { + "bytes": 31035526, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/test_batch", + "sha256": "f53d8d457504f7cff4ea9e021afcf0e0ad8e24a91f3fc42091b8adef61157831" + } + ], + "dataset": "cifar10", + "evaluation_split": "validation", + "input_layout": "NCHW", + "input_shape": [ + 3, + 32, + 32 + ], + "loader_seed": 0, + "normalization_mean": [ + 0.49140000343322754, + 0.4821999967098236, + 0.4465000033378601 + ], + "normalization_std": [ + 0.24699999392032623, + 0.2434999942779541, + 0.26159998774528503 + ], + "split_from_training_only": true, + "split_seed": 2027, + "test_examples": 10000, + "train_examples": 10000, + "training_augmentation": "random_crop_32_padding_4_zero_then_horizontal_flip_p0.5", + "validation_class_counts": { + "0": 500, + "1": 500, + "2": 500, + "3": 500, + "4": 500, + "5": 500, + "6": 500, + "7": 500, + "8": 500, + "9": 500 + }, + "validation_examples": 5000, + "validation_index_sha256": "8328b206a97c420e49e54e3eca4abe3274c4756b084355784ea3fb8059e4515b" + }, + "timing": { + "apical_warmup_wall_s": 0.0, + "evaluation_wall_s": 0.41851377487182617, + "mirror_warmup_wall_s": 0.6023821830749512, + "predictor_warmup_wall_s": 0.0, + "timing_excludes_data_loading_hashing_and_model_construction": true, + "total_timed_wall_s": 1.0211255550384521, + "train_wall_s": 0.0 + }, + "work": { + "apical_macs_per_example": 40108672, + "causal_scalar_observations": 0, + "components": { + "apical_projection_macs": 0, + "apical_regression_macs": 0, + "bp_reverse_macs_estimate": 0, + "local_weight_correlation_macs": 0, + "mirror_feedback_prediction_macs": 815267840, + "mirror_local_correlation_macs": 815267840, + "mirror_response_macs": 815267840, + "ordinary_forward_macs": 0, + "perturbation_forward_macs": 0, + "warmup_clean_forward_macs": 0 + }, + "definition": "multiply-accumulates in conv/linear maps; one local weight correlation equals one forward-weight MAC count; BP reverse is estimated as one weight-gradient plus one activation-gradient convolution per forward convolution; elementwise nonlinearities and optimizer arithmetic excluded", + "forward_macs_per_example": 40551040, + "logical_batch_loss_queries": 0, + "mirror_probe_examples": 20, + "mirror_readout_probe_examples": 20480, + "per_example_cross_entropy_terms": 0, + "total_clean_forward_examples": 0, + "total_forward_equivalent_examples": 0, + "total_macs_estimate": 2445803520 + } +} diff --git a/results/residual_mirror_capture/rrm_etaM0.3.json b/results/residual_mirror_capture/rrm_etaM0.3.json new file mode 100644 index 0000000..d0a5ebc --- /dev/null +++ b/results/residual_mirror_capture/rrm_etaM0.3.json @@ -0,0 +1,455 @@ +{ + "architecture": { + "adaptive_apical_parameters": 0, + "adaptive_feedback_parameters": 267904, + "base_width": 16, + "blocks_per_stage": 3, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "depth": 20, + "family": "CIFAR 6n+2 ResNet, option-A shortcuts", + "fixed_feedback_parameters": 0, + "fixed_traffic_coefficients": 0, + "forward_parameters": 269722, + "hidden_shapes": [ + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 16, + 32, + 32 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 32, + 16, + 16 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ], + [ + 64, + 8, + 8 + ] + ], + "normalization": "batchnorm", + "predictor_parameters": 0, + "residual_scale": 1.0, + "vectorizer_mode": null, + "vectorizer_parameters": 0 + }, + "args": { + "a_scale": 1.0, + "a_warmup_steps": 0, + "alignment_probe": 64, + "apical_calibration_mode": "unit_targets", + "apical_seed": null, + "augment_train": 1, + "batch_size": 128, + "bn_eps": 1e-05, + "bn_momentum": 0.1, + "data_dir": "/home/yurenh2/sdrn/data", + "depth": 20, + "device": "cuda", + "epochs": 0, + "eta_A": 0.01, + "eta_P": 0.01, + "eval_every": 0, + "eval_split": "validation", + "learn_P": 0, + "loader_seed": 0, + "lr": 0.1, + "lr_gamma": 0.1, + "lr_milestones": "100,150", + "lr_schedule": "constant", + "max_steps": 0, + "mirror_batch_size": 1, + "mirror_eta": 0.3, + "mirror_every": 16, + "mirror_noise_std": 1.0, + "mirror_seed": 3000, + "mirror_warmup_steps": 20, + "mode": "rrm", + "momentum": 0.9, + "normalization": "batchnorm", + "nuisance_scale": 0.0, + "out": "results/residual_mirror_capture/rrm_etaM0.3.json", + "output_lr": 0.1, + "pert_directions": 1, + "pert_every": 4, + "pert_sigma": 0.01, + "perturb_seed": 1000, + "predictor_warmup_steps": 0, + "residual_scale": null, + "seed": 0, + "split_seed": 2027, + "train_limit": 10000, + "use_residual": 1, + "val_examples": 5000, + "vectorizer_mode": "spatial_template", + "warmup_epochs": 0, + "weight_decay": 0.0001, + "weight_scale": 1.0, + "width": 16 + }, + "calibration_metric_space": "local_parent_child_response_residual", + "counters": { + "apical_warmup_examples": 0, + "calibration_event_examples": 0, + "causal_scalar_observations": 0, + "logical_batch_loss_queries": 0, + "mirror_conv_examples": 20, + "mirror_events": 20, + "mirror_readout_examples": 20480, + "ordinary_examples": 0, + "per_example_loss_terms": 0, + "perturbation_events": 0, + "perturbation_forward_examples": 0, + "predictor_warmup_examples": 0 + }, + "diagnostics": { + "early_third_mean": 0.00348945043515414, + "feedback_forward_cosine": [ + 0.9999993443489075, + 0.9999992251396179, + 0.9999993443489075, + 0.9999993443489075, + 0.9999993443489075, + 0.9999992847442627, + 0.9999954104423523, + 0.999967098236084, + 0.9999670386314392, + 0.9999639391899109, + 0.999970555305481, + 0.9999678730964661, + 0.8819347023963928, + 0.04425355792045593, + 0.03485032171010971, + 0.04099895805120468, + 0.033140137791633606, + 0.03251386433839798, + 0.9999995827674866 + ], + "feedback_forward_norm_ratio": [ + 0.9992052912712097, + 0.999176025390625, + 0.9992217421531677, + 0.9992625713348389, + 0.9992009401321411, + 0.9992427825927734, + 0.9992942214012146, + 0.999183177947998, + 0.9992207884788513, + 0.9991571307182312, + 0.9993154406547546, + 0.9994397759437561, + 1.1370186805725098, + 27.31791877746582, + 27.16834831237793, + 26.48126792907715, + 29.63273048400879, + 30.131799697875977, + 0.9992100596427917 + ], + "innovation_negative_gradient_cosine": [ + 0.003967876546084881, + 0.0034768409095704556, + 0.005226013250648975, + 0.00637441873550415, + 0.00127668387722224, + 0.0006148692918941379, + 0.0032356157898902893, + 0.0040656100027263165, + 0.003892980283126235, + 0.006772768218070269, + 0.007718275301158428, + 0.009753046557307243, + 0.00867580994963646, + 0.004539509303867817, + -0.005354281514883041, + -0.02212643437087536, + 0.008024247363209724, + -0.029004445299506187, + 0.9999995231628418 + ], + "normalization_state": "training_batch_stats_without_running_update", + "raw_negative_gradient_cosine": [ + 0.003967876546084881, + 0.0034768409095704556, + 0.005226013250648975, + 0.00637441873550415, + 0.00127668387722224, + 0.0006148692918941379, + 0.0032356157898902893, + 0.0040656100027263165, + 0.003892980283126235, + 0.006772768218070269, + 0.007718275301158428, + 0.009753046557307243, + 0.00867580994963646, + 0.004539509303867817, + -0.005354281514883041, + -0.02212643437087536, + 0.008024247363209724, + -0.029004445299506187, + 0.9999995231628418 + ], + "teaching_negative_gradient_cosine": [ + 0.003967876546084881, + 0.0034768409095704556, + 0.005226013250648975, + 0.00637441873550415, + 0.00127668387722224, + 0.0006148692918941379, + 0.0032356157898902893, + 0.0040656100027263165, + 0.003892980283126235, + 0.006772768218070269, + 0.007718275301158428, + 0.009753046557307243, + 0.00867580994963646, + 0.004539509303867817, + -0.005354281514883041, + -0.02212643437087536, + 0.008024247363209724, + -0.029004445299506187, + 0.9999995231628418 + ], + "wall_s": 0.13489723205566406 + }, + "epochs": [], + "evaluation_protocol": { + "test_evaluations": 0, + "test_used_for_selection": false, + "validation_evaluations": 1 + }, + "final": { + "accuracy": 0.1004, + "epoch": 0, + "evaluation_split": "validation", + "finite": true, + "loss": 16.593759375, + "step": 0 + }, + "hardware": { + "cuda_device_name": "NVIDIA GeForce GTX 1080", + "cuda_visible_devices": "7", + "device": "cuda", + "device_total_memory_bytes": 8507949056, + "peak_memory_allocated_bytes": 1730634752, + "peak_memory_reserved_bytes": 2181038080, + "torch_version": "2.3.1+cu118" + }, + "mirror_warmup": { + "first": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 1.4124661316345333, + "mirror_response_residual_rms": 1.90927576279183, + "mirror_update_rms": 0.25520769706611623, + "readout_batch_size": 1024 + }, + "last": { + "conv_batch_size": 1, + "mirror_response_residual_fraction": 7.735358358942393, + "mirror_response_residual_rms": 10.475033440813764, + "mirror_update_rms": 3.861606219890987, + "readout_batch_size": 1024 + }, + "mean": { + "conv_batch_size": 1.0, + "mirror_response_residual_fraction": 2.7695583682252174, + "mirror_response_residual_rms": 3.750395332923587, + "mirror_update_rms": 1.3223578319202012, + "readout_batch_size": 1024.0 + }, + "steps": 20 + }, + "protocol_family": "oral_a_cifar_local_resnet_development", + "provenance": { + "git_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "git_tracked_dirty": false + }, + "schema_version": 1, + "split": { + "cifar_source_files": [ + { + "bytes": 31035704, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_1", + "sha256": "54636561a3ce25bd3e19253c6b0d8538147b0ae398331ac4a2d86c6d987368cd" + }, + { + "bytes": 31035320, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_2", + "sha256": "766b2cef9fbc745cf056b3152224f7cf77163b330ea9a15f9392beb8b89bc5a8" + }, + { + "bytes": 31035999, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_3", + "sha256": "0f00d98ebfb30b3ec0ad19f9756dc2630b89003e10525f5e148445e82aa6a1f9" + }, + { + "bytes": 31035696, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_4", + "sha256": "3f7bb240661948b8f4d53e36ec720d8306f5668bd0071dcb4e6c947f78e9682b" + }, + { + "bytes": 31035623, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/data_batch_5", + "sha256": "d91802434d8376bbaeeadf58a737e3a1b12ac839077e931237e0dcd43adcb154" + }, + { + "bytes": 31035526, + "path": "/home/yurenh2/sdrn/data/cifar-10-batches-py/test_batch", + "sha256": "f53d8d457504f7cff4ea9e021afcf0e0ad8e24a91f3fc42091b8adef61157831" + } + ], + "dataset": "cifar10", + "evaluation_split": "validation", + "input_layout": "NCHW", + "input_shape": [ + 3, + 32, + 32 + ], + "loader_seed": 0, + "normalization_mean": [ + 0.49140000343322754, + 0.4821999967098236, + 0.4465000033378601 + ], + "normalization_std": [ + 0.24699999392032623, + 0.2434999942779541, + 0.26159998774528503 + ], + "split_from_training_only": true, + "split_seed": 2027, + "test_examples": 10000, + "train_examples": 10000, + "training_augmentation": "random_crop_32_padding_4_zero_then_horizontal_flip_p0.5", + "validation_class_counts": { + "0": 500, + "1": 500, + "2": 500, + "3": 500, + "4": 500, + "5": 500, + "6": 500, + "7": 500, + "8": 500, + "9": 500 + }, + "validation_examples": 5000, + "validation_index_sha256": "8328b206a97c420e49e54e3eca4abe3274c4756b084355784ea3fb8059e4515b" + }, + "timing": { + "apical_warmup_wall_s": 0.0, + "evaluation_wall_s": 0.42837023735046387, + "mirror_warmup_wall_s": 0.52046799659729, + "predictor_warmup_wall_s": 0.0, + "timing_excludes_data_loading_hashing_and_model_construction": true, + "total_timed_wall_s": 0.9490752220153809, + "train_wall_s": 0.0 + }, + "work": { + "apical_macs_per_example": 40108672, + "causal_scalar_observations": 0, + "components": { + "apical_projection_macs": 0, + "apical_regression_macs": 0, + "bp_reverse_macs_estimate": 0, + "local_weight_correlation_macs": 0, + "mirror_feedback_prediction_macs": 815267840, + "mirror_local_correlation_macs": 815267840, + "mirror_response_macs": 815267840, + "ordinary_forward_macs": 0, + "perturbation_forward_macs": 0, + "warmup_clean_forward_macs": 0 + }, + "definition": "multiply-accumulates in conv/linear maps; one local weight correlation equals one forward-weight MAC count; BP reverse is estimated as one weight-gradient plus one activation-gradient convolution per forward convolution; elementwise nonlinearities and optimizer arithmetic excluded", + "forward_macs_per_example": 40551040, + "logical_batch_loss_queries": 0, + "mirror_probe_examples": 20, + "mirror_readout_probe_examples": 20480, + "per_example_cross_entropy_terms": 0, + "total_clean_forward_examples": 0, + "total_forward_equivalent_examples": 0, + "total_macs_estimate": 2445803520 + } +} diff --git a/results/residual_mirror_capture_gate.json b/results/residual_mirror_capture_gate.json new file mode 100644 index 0000000..633247b --- /dev/null +++ b/results/residual_mirror_capture_gate.json @@ -0,0 +1,133 @@ +{ + "checks": { + "all_four_records_finite": true, + "all_layer_at_least_0.70": true, + "candidate_selected": true, + "early_third_at_least_0.65": true, + "feedback_norm_ratios_in_0.5_to_1.5": true, + "mean_feedback_forward_cosine_at_least_0.93": true, + "zero_task_loss_queries": true + }, + "confirmation_test_seeds_touched": false, + "matched_fixed_hfa": { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.011278870683255312, + "early_third_alignment": -0.00026315699263553444, + "max_feedback_forward_norm_ratio": 1.0550107955932617, + "mean_feedback_forward_cosine": 0.0016142042954207252, + "min_feedback_forward_norm_ratio": 0.9593725800514221 + }, + "mirror_eta": null, + "mirror_events": 0, + "mode": "hfa", + "path": "results/residual_mirror_capture/fixed_hfa.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 0 + }, + "protocol": "residual_response_mirror_capture_v1", + "review_score_after": 5, + "review_score_before": 5, + "rows": [ + { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.011278870683255312, + "early_third_alignment": -0.00026315699263553444, + "max_feedback_forward_norm_ratio": 1.0550107955932617, + "mean_feedback_forward_cosine": 0.0016142042954207252, + "min_feedback_forward_norm_ratio": 0.9593725800514221 + }, + "mirror_eta": null, + "mirror_events": 0, + "mode": "hfa", + "path": "results/residual_mirror_capture/fixed_hfa.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 0 + }, + { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.23500083033975802, + "early_third_alignment": 0.07501635327935219, + "max_feedback_forward_norm_ratio": 0.7924403548240662, + "mean_feedback_forward_cosine": 0.6229652448704368, + "mean_mirror_update_rms": 0.19879612560335747, + "mean_response_residual_fraction": 1.0790779544942406, + "mean_response_residual_rms": 1.461456808634224, + "min_feedback_forward_norm_ratio": 0.6949356198310852 + }, + "mirror_eta": 0.03, + "mirror_events": 20, + "mode": "rrm", + "path": "results/residual_mirror_capture/rrm_etaM0.03.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 2445803520 + }, + { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.7302531662740206, + "early_third_alignment": 0.6650698681672415, + "max_feedback_forward_norm_ratio": 1.0179650783538818, + "mean_feedback_forward_cosine": 0.9536802360886022, + "mean_mirror_update_rms": 0.15685383656211757, + "mean_response_residual_fraction": 0.6719071790962137, + "mean_response_residual_rms": 0.9100978838204868, + "min_feedback_forward_norm_ratio": 0.8841801881790161 + }, + "mirror_eta": 0.1, + "mirror_events": 20, + "mode": "rrm", + "path": "results/residual_mirror_capture/rrm_etaM0.1.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 2445803520 + }, + { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.05374362775565762, + "early_third_alignment": 0.00348945043515414, + "max_feedback_forward_norm_ratio": 30.131799697875977, + "mean_feedback_forward_cosine": 0.7403957330082592, + "mean_mirror_update_rms": 1.3223578319202012, + "mean_response_residual_fraction": 2.7695583682252174, + "mean_response_residual_rms": 3.750395332923587, + "min_feedback_forward_norm_ratio": 0.9991571307182312 + }, + "mirror_eta": 0.3, + "mirror_events": 20, + "mode": "rrm", + "path": "results/residual_mirror_capture/rrm_etaM0.3.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 2445803520 + } + ], + "score_change_rule": "inherited baseline capture cannot raise score", + "selected_rrm": { + "finite": true, + "logical_batch_loss_queries": 0, + "metrics": { + "all_layer_alignment": 0.7302531662740206, + "early_third_alignment": 0.6650698681672415, + "max_feedback_forward_norm_ratio": 1.0179650783538818, + "mean_feedback_forward_cosine": 0.9536802360886022, + "mean_mirror_update_rms": 0.15685383656211757, + "mean_response_residual_fraction": 0.6719071790962137, + "mean_response_residual_rms": 0.9100978838204868, + "min_feedback_forward_norm_ratio": 0.8841801881790161 + }, + "mirror_eta": 0.1, + "mirror_events": 20, + "mode": "rrm", + "path": "results/residual_mirror_capture/rrm_etaM0.1.json", + "source_commit": "18aff5b55e3ad1f6e244ad25a8eae44f25f4e5c1", + "total_macs": 2445803520 + }, + "status": "passed" +} |