diff options
| -rw-r--r-- | BASELINES.md | 5 | ||||
| -rw-r--r-- | NOVELTY.md | 6 | ||||
| -rw-r--r-- | README.md | 9 | ||||
| -rw-r--r-- | THEORY.md | 38 | ||||
| -rw-r--r-- | experiments/conv_local_smoke.py | 49 | ||||
| -rw-r--r-- | experiments/conv_run.py | 84 | ||||
| -rw-r--r-- | sdil/conv.py | 120 |
7 files changed, 306 insertions, 5 deletions
diff --git a/BASELINES.md b/BASELINES.md index 6b76d09..7cc6c10 100644 --- a/BASELINES.md +++ b/BASELINES.md @@ -22,6 +22,11 @@ The audited implementations and completed results are in `RESULTS.md`: hierarchical extension. Its frozen validation-only screen is specified in `HFA_BASELINE.md`; the selected short endpoint is 43.52% versus matched DFA's 37.16%, but its 50% full-run gate failed; +- normalized response mirroring (`wm`) is an inherited weight-mirror/weight- + estimation control: Gaussian parent probes pass through ordinary local + forward synapses, and a separate update consumes only parent/child activities + to estimate each feedback kernel. It is deliberately treated as a strong + learned-FA baseline, not as SDIL novelty; - direct node perturbation uses a causal target on every hidden update and no learned vectorizer; - the no-traffic, `P=0` SDIL backbone is learned direct feedback by node @@ -65,6 +65,12 @@ not by itself make recursive learned feedback novel. It is treated as a credit-path engineering component unless the complete innovation model shows a load-bearing residualization result beyond those mandatory comparators. +Likewise, the normalized local response-mirror path is explicitly inherited +from weight mirroring/weight estimation. Even if it recovers BP-level scale, +that performance belongs to the baseline. A paper-level SDIL gain requires the +somato-dendritic innovation subtraction to remain necessary on top of the +mirrored hierarchical path under a frozen mixed-traffic intervention. + ## Candidate SDIL contribution The candidate contribution is the combination @@ -130,6 +130,15 @@ child/parent correlation. Its BatchNorm-coupled JVP and symmetric-limit delta rule are executable smoke tests. `ORAL_A_V4.md` freezes the causal-capture and conditional accuracy gates; no accuracy claim is attached until they pass. +Because the global-scalar V4 gate failed, the next strong inherited comparator +uses normalized local response mirroring. Its observation phase sends Gaussian +parent activity through ordinary local forward synapses; a separate feedback +update sees only the probe/child-response pairs. The smoke suite verifies both +high finite-sample kernel recovery and that changing forward parameters after +observation cannot affect the update. Any scale obtained this way is credited +to the weight-mirror baseline until innovation residualization is shown to be +load-bearing on top of it. + The subsequent V3 mechanism estimates the required A/G matrix statistics directly by perturbing the vectorizer parameter subspace. It remains forward-only and uses two causal loss queries per direction. Its exact and @@ -253,6 +253,44 @@ the stable rate left early alignment near zero, while larger rates caused a does not imply usable finite-budget signal in the 267,904-dimensional joint parameter space. +### Local response mirroring as an inherited strong baseline + +V4's failure motivates changing the observation, not merely its rate. The +weight-mirror identity of Akrout et al. starts from white local parent noise +`z` and its child response `u=Wz`: + +```text +E[z u^T] = sigma^2 W^T. +``` + +For convolutions, the corresponding local weight correlation divided by the +number of valid spatial probe pairs and `sigma^2` is an unbiased estimator of +the forward kernel. The normalized response-mirror baseline uses a +bias-blocked pre-BatchNorm observation phase and updates + +```text +What_e = corr(z_e, u_e) / (count_e sigma^2), +Q_e <- (1-eta_M) Q_e + eta_M What_e. +``` + +The readout is identical, with the sign reversed to match this repository's +negative-gradient teaching convention. Observation generation uses the +ordinary local forward synapses; the Q/R update API receives only probe and +response activities. An executable independence audit changes every forward +parameter after observation and verifies that the feedback update is bitwise +unchanged. + +This is a stabilized system-identification variant of +[weight mirroring](https://proceedings.neurips.cc/paper_files/paper/2019/file/f387624df552cea2f369918c5e1e12bc-Paper.pdf), +not an SDIL contribution. The original Hebbian-plus-decay rule can be highly +metaparameter-sensitive, as analyzed by +[Kunin et al.](https://proceedings.mlr.press/v119/kunin20a/kunin20a.pdf); +normalizing the local sufficient statistic makes the estimator's scale +explicit. At the deterministic smoke setting, one 16-probe observation has +mean/minimum feedback-forward cosine `0.9887/0.9661`, norm ratios +`[0.9537,1.0444]`, and zero forward-parameter access error in the update. +These are mechanics, not task evidence. A frozen comparator gate is required. + ## 2. What is sufficient for a descent step Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let diff --git a/experiments/conv_local_smoke.py b/experiments/conv_local_smoke.py index 894f9d3..7d24154 100644 --- a/experiments/conv_local_smoke.py +++ b/experiments/conv_local_smoke.py @@ -12,7 +12,9 @@ from sdil.conv import (CIFARHierarchicalFAResNet, CIFARLocalResNet, CIFARSDILResNet, ConvSDILConfig, channel_subspace_apical_calibration, conv_hierarchical_step, conv_local_step, + hierarchical_mirror_observations, hierarchical_parameter_subspace_calibration, + normalized_response_mirror_update, simultaneous_conv_node_perturbation, vectorizer_subspace_apical_calibration) @@ -756,6 +758,52 @@ def hierarchical_parameter_calibration_checks(): } +def normalized_response_mirror_checks(): + """Audit local response estimation and absence of W access in the update.""" + net = CIFARHierarchicalFAResNet( + depth=8, base_width=2, seed=121, dtype=torch.float64, + normalization="batchnorm") + observations = hierarchical_mirror_observations( + net, batch_size=16, noise_std=1.0, + generator=torch.Generator().manual_seed(122)) + metrics, _ = normalized_response_mirror_update( + net, observations, eta=1.0) + pairs = list(zip(net.Q[1:], net.W[1:])) + [ + (net.R_out, -net.W_out.t())] + cosines = [float(F.cosine_similarity( + feedback.flatten(), target.flatten(), dim=0)) + for feedback, target in pairs] + norm_ratios = [float(feedback.norm() / target.norm()) + for feedback, target in pairs] + assert sum(cosines) / len(cosines) > 0.985 + assert min(cosines) > 0.95 + assert min(norm_ratios) > 0.90 and max(norm_ratios) < 1.10 + + # The update consumes observations only. Changing every forward parameter + # after those observations were generated must not change the Q/R update. + left = CIFARHierarchicalFAResNet( + depth=8, base_width=2, seed=123, dtype=torch.float64) + right = CIFARHierarchicalFAResNet( + depth=8, base_width=2, seed=123, dtype=torch.float64) + shared_observations = hierarchical_mirror_observations( + left, batch_size=2, generator=torch.Generator().manual_seed(124)) + for value in right.W + [right.W_out]: + value.normal_(generator=torch.Generator().manual_seed(value.numel())) + normalized_response_mirror_update(left, shared_observations, eta=0.2) + normalized_response_mirror_update(right, shared_observations, eta=0.2) + independence_error = max(float((a - b).abs().max()) for a, b in zip( + left.Q[1:] + [left.R_out], right.Q[1:] + [right.R_out])) + assert independence_error == 0.0 + return { + "mirror_estimate_mean_forward_cosine": sum(cosines) / len(cosines), + "mirror_estimate_min_forward_cosine": min(cosines), + "mirror_estimate_min_norm_ratio": min(norm_ratios), + "mirror_estimate_max_norm_ratio": max(norm_ratios), + "mirror_update_forward_parameter_independence_error": independence_error, + "mirror_update_rms": metrics["mirror_update_rms"], + } + + def apical_learning_checks(): torch.manual_seed(11) net = CIFARSDILResNet(depth=8, base_width=2, seed=6) @@ -863,6 +911,7 @@ def main(): report.update(vectorizer_subspace_estimator_check()) report.update(hierarchical_feedback_checks()) report.update(hierarchical_parameter_calibration_checks()) + report.update(normalized_response_mirror_checks()) report.update(apical_learning_checks()) print(report) print("ALL CONVOLUTIONAL LOCAL-ELIGIBILITY CHECKS PASSED") diff --git a/experiments/conv_run.py b/experiments/conv_run.py index fb759b4..fcb6690 100644 --- a/experiments/conv_run.py +++ b/experiments/conv_run.py @@ -19,6 +19,7 @@ from sdil.conv import (CIFARHierarchicalFAResNet, CIFARLocalResNet, conv_hierarchical_step, conv_learned_hierarchical_step, conv_local_step, evaluate_conv, hierarchical_parameter_subspace_calibration) +from sdil.conv import normalized_response_mirror_step from sdil.data import DATA_DIR, get_cifar_image_splits @@ -105,7 +106,7 @@ def build(args): bn_momentum=args.bn_momentum, bn_eps=args.bn_eps) if args.mode == "bp": return CIFARLocalResNet(**common), None - if args.mode in ("hfa", "lhfa"): + if args.mode in ("hfa", "lhfa", "wm"): net = CIFARHierarchicalFAResNet( **common, feedback_seed=args.apical_seed, feedback_scale=args.a_scale) @@ -156,6 +157,11 @@ def work_report(net, mode, counters): apical_regression = (regression_multiplier * counters["calibration_event_examples"] * apical_macs) + mirror_conv_macs = max(0, apical_macs - getattr(net, "R_out", torch.empty(0)).numel()) + mirror_readout_macs = getattr(net, "R_out", torch.empty(0)).numel() + mirror_forward = (counters["mirror_conv_examples"] * mirror_conv_macs + + counters["mirror_readout_examples"] * mirror_readout_macs) + mirror_correlation = mirror_forward components = { "ordinary_forward_macs": normal_forward, "warmup_clean_forward_macs": warmup_forward, @@ -164,6 +170,8 @@ def work_report(net, mode, counters): "local_weight_correlation_macs": local_correlation, "apical_projection_macs": apical_inference, "apical_regression_macs": apical_regression, + "mirror_response_macs": mirror_forward, + "mirror_local_correlation_macs": mirror_correlation, } return { "forward_macs_per_example": forward_macs, @@ -180,6 +188,8 @@ def work_report(net, mode, counters): "logical_batch_loss_queries": counters["logical_batch_loss_queries"], "causal_scalar_observations": counters["causal_scalar_observations"], "per_example_cross_entropy_terms": counters["per_example_loss_terms"], + "mirror_probe_examples": counters["mirror_conv_examples"], + "mirror_readout_probe_examples": counters["mirror_readout_examples"], "definition": ( "multiply-accumulates in conv/linear maps; one local weight correlation " "equals one forward-weight MAC count; BP reverse is estimated as one " @@ -196,6 +206,12 @@ def run(args): raise ValueError("apical/predictor warmup is restricted to SDIL") if args.mode == "lhfa" and args.learn_P: raise ValueError("predictor learning is not defined for learned HFA") + if args.mode != "wm" and args.mirror_warmup_steps: + raise ValueError("mirror warmup is restricted to weight mirror mode") + if args.mirror_every < 1 or args.mirror_batch_size < 1: + raise ValueError("invalid mirror cadence or batch size") + if not 0.0 < args.mirror_eta <= 1.0 or args.mirror_noise_std <= 0: + raise ValueError("invalid mirror learning hyperparameters") torch.manual_seed(args.seed) if str(args.device).startswith("cuda"): if not torch.cuda.is_available(): @@ -221,6 +237,8 @@ def run(args): args.perturb_seed) warmup_generator = torch.Generator(device=torch.device(args.device)).manual_seed( args.perturb_seed + 1) + mirror_generator = torch.Generator(device=torch.device(args.device)).manual_seed( + args.mirror_seed) counters = { "ordinary_examples": 0, @@ -232,6 +250,9 @@ def run(args): "causal_scalar_observations": 0, "per_example_loss_terms": 0, "perturbation_events": 0, + "mirror_conv_examples": 0, + "mirror_readout_examples": 0, + "mirror_events": 0, } log = { "schema_version": 1, @@ -239,10 +260,12 @@ def run(args): "calibration_metric_space": ( None if config is None or args.mode == "hfa" else "hierarchical_feedback_parameters" if args.mode == "lhfa" else { + "wm": "local_parent_child_response", "unit_targets": "full_hidden_field", "channel_subspace": "channel_basis_moments", "vectorizer_subspace": "vectorizer_parameter_gradients", - }[config.apical_calibration_mode]), + }[args.mode if args.mode == "wm" + else config.apical_calibration_mode]), "args": vars(args), "provenance": provenance(), "split": split, @@ -268,7 +291,7 @@ def run(args): if args.mode == "hfa" else 0), "adaptive_feedback_parameters": ( getattr(net, "n_fixed_feedback_parameters", 0) - if args.mode == "lhfa" else 0), + if args.mode in ("lhfa", "wm") else 0), }, "epochs": [], } @@ -277,7 +300,30 @@ def run(args): total_start = time.time() predictor_warmup_wall = 0.0 apical_warmup_wall = 0.0 + mirror_warmup_wall = 0.0 loader_state = train.g.get_state().clone() + if args.mode == "wm" and args.mirror_warmup_steps: + sync(args.device) + mirror_start = time.time() + mirror_metrics = [] + for _ in range(args.mirror_warmup_steps): + metric = normalized_response_mirror_step( + net, batch_size=args.mirror_batch_size, + noise_std=args.mirror_noise_std, eta=args.mirror_eta, + generator=mirror_generator) + mirror_metrics.append(metric) + counters["mirror_conv_examples"] += args.mirror_batch_size + counters["mirror_readout_examples"] += metric["readout_batch_size"] + counters["mirror_events"] += 1 + log["mirror_warmup"] = { + "steps": args.mirror_warmup_steps, + "first": mirror_metrics[0], + "mean": {key: sum(value[key] for value in mirror_metrics) + / len(mirror_metrics) for key in mirror_metrics[0]}, + "last": mirror_metrics[-1], + } + sync(args.device) + mirror_warmup_wall = time.time() - mirror_start if config is not None and config.learn_P and args.predictor_warmup_steps: sync(args.device) warmup_start = time.time() @@ -365,6 +411,7 @@ def run(args): loss_sum = 0.0 examples = 0 calibration_metrics = [] + mirror_metrics = [] for x, y in train: batch = x.shape[0] if args.mode == "bp": @@ -383,6 +430,20 @@ def run(args): did_perturb = result["did_perturb"] if result["calibration"] is not None: calibration_metrics.append(result["calibration"]) + elif args.mode == "wm": + if step % args.mirror_every == 0: + mirror_metric = normalized_response_mirror_step( + net, batch_size=args.mirror_batch_size, + noise_std=args.mirror_noise_std, eta=args.mirror_eta, + generator=mirror_generator) + mirror_metrics.append(mirror_metric) + counters["mirror_conv_examples"] += args.mirror_batch_size + counters["mirror_readout_examples"] += mirror_metric[ + "readout_batch_size"] + counters["mirror_events"] += 1 + result = conv_hierarchical_step(net, x, y, config) + loss = result["loss"] + did_perturb = False else: result = conv_local_step( net, x, y, config, step, generator=perturb_generator) @@ -423,6 +484,11 @@ def run(args): / len(calibration_metrics) for key in calibration_metrics[0] } + if mirror_metrics: + record["mirror"] = { + key: sum(value[key] for value in mirror_metrics) + / len(mirror_metrics) for key in mirror_metrics[0] + } if args.eval_every and (epoch + 1) % args.eval_every == 0: sync(args.device) eval_start = time.time() @@ -461,7 +527,7 @@ def run(args): diagnostic_start = time.time() diagnostics = (conv_hierarchical_alignment_report( net, train.x[:probe], train.y[:probe]) - if args.mode in ("hfa", "lhfa") else + if args.mode in ("hfa", "lhfa", "wm") else conv_alignment_report( net, train.x[:probe], train.y[:probe], config)) sync(args.device) @@ -477,6 +543,7 @@ def run(args): "timing": { "predictor_warmup_wall_s": predictor_warmup_wall, "apical_warmup_wall_s": apical_warmup_wall, + "mirror_warmup_wall_s": mirror_warmup_wall, "train_wall_s": train_wall, "evaluation_wall_s": eval_wall, "total_timed_wall_s": total_wall, @@ -508,7 +575,8 @@ def run(args): def parse_args(): parser = argparse.ArgumentParser() parser.add_argument( - "--mode", choices=("bp", "dfa", "hfa", "lhfa", "sdil", "nodepert"), + "--mode", choices=( + "bp", "dfa", "hfa", "lhfa", "wm", "sdil", "nodepert"), required=True) parser.add_argument("--out", required=True) parser.add_argument("--device", default="cpu") @@ -561,6 +629,12 @@ def parse_args(): default="unit_targets") parser.add_argument("--predictor_warmup_steps", type=int, default=0) parser.add_argument("--a_warmup_steps", type=int, default=0) + parser.add_argument("--mirror_warmup_steps", type=int, default=0) + parser.add_argument("--mirror_every", type=int, default=16) + parser.add_argument("--mirror_batch_size", type=int, default=1) + parser.add_argument("--mirror_eta", type=float, default=0.1) + parser.add_argument("--mirror_noise_std", type=float, default=1.0) + parser.add_argument("--mirror_seed", type=int, default=3000) parser.add_argument("--alignment_probe", type=int, default=0) return parser.parse_args() diff --git a/sdil/conv.py b/sdil/conv.py index ba4e980..cc373e8 100644 --- a/sdil/conv.py +++ b/sdil/conv.py @@ -661,6 +661,126 @@ def hierarchical_parameter_subspace_calibration( return calibration +@torch.no_grad() +def hierarchical_mirror_observations(net, batch_size=1, noise_std=1.0, + generator=None): + """Generate local bias-blocked probe/child-response pairs. + + This is the observation phase of a normalized weight-mirror baseline. A + parent population emits independent Gaussian noise and the ordinary + forward synapses generate the child's preactivation response. The return + value contains activities only; the subsequent feedback update has no + access to forward parameters. + """ + if not isinstance(net, CIFARHierarchicalFAResNet): + raise TypeError("mirror observations require a hierarchical net") + if batch_size < 1 or noise_std <= 0: + raise ValueError("invalid mirror observation hyperparameters") + if generator is None: + generator = torch.Generator(device=net.W_out.device).manual_seed(0) + conv = [None] + for index in range(1, len(net.W)): + weight = net.W[index] + spec = net.layer_specs[index] + _, in_channels, _, _ = weight.shape + _, out_height, out_width = spec.hidden_shape + input_shape = ( + batch_size, in_channels, + out_height * spec.stride, out_width * spec.stride) + probe = torch.randn( + input_shape, generator=generator, device=weight.device, + dtype=weight.dtype).mul_(noise_std) + response = F.conv2d( + probe, weight, stride=spec.stride, padding=spec.padding) + conv.append((probe, response)) + # The dense readout has no spatial sample multiplicity. Use a local probe + # population large enough to estimate its channel covariance accurately; + # its MAC cost is recorded explicitly by the runner. + readout_batch = max( + batch_size, 16 * net.W_out.shape[1], 16 * net.W_out.shape[0]) + readout_probe = torch.randn( + readout_batch, net.W_out.shape[1], generator=generator, + device=net.W_out.device, dtype=net.W_out.dtype).mul_(noise_std) + readout_response = readout_probe @ net.W_out.t() + return { + "conv": conv, + "readout": (readout_probe, readout_response), + "noise_variance": noise_std ** 2, + "conv_batch_size": batch_size, + "readout_batch_size": readout_batch, + } + + +@torch.no_grad() +def normalized_response_mirror_update(net, observations, eta=0.1): + """Update Q/R from local probe/response pairs without reading W. + + For white parent noise ``z`` and a bias-blocked child response ``u=Wz``, + the normalized local correlation is an unbiased estimator of ``W``. An + exponential delta rule makes Q track that estimate. This is a stabilized + weight-estimation baseline inherited from weight-mirror work, not an SDIL + contribution. + """ + if not isinstance(net, CIFARHierarchicalFAResNet): + raise TypeError("mirror update requires a hierarchical net") + if not 0.0 < eta <= 1.0: + raise ValueError("mirror eta must be in (0, 1]") + variance = float(observations.get("noise_variance", 0.0)) + conv = observations.get("conv") + if variance <= 0 or conv is None or len(conv) != len(net.Q): + raise ValueError("invalid mirror observations") + estimates = [None] + update_power = 0.0 + estimate_power = 0.0 + parameters = 0 + for index in range(1, len(net.Q)): + probe, response = conv[index] + spec = net.layer_specs[index] + if response.shape[1] != net.Q[index].shape[0]: + raise ValueError("mirror child response channel mismatch") + correlation = torch.nn.grad.conv2d_weight( + probe, net.Q[index].shape, response, + stride=spec.stride, padding=spec.padding) + counts = torch.nn.grad.conv2d_weight( + torch.ones_like(probe), net.Q[index].shape, + torch.ones_like(response), stride=spec.stride, + padding=spec.padding) + estimate = correlation / (variance * counts.clamp_min(1.0)) + update = estimate - net.Q[index] + net.Q[index].add_(update, alpha=eta) + estimates.append(estimate) + update_power += float(update.square().sum()) + estimate_power += float(estimate.square().sum()) + parameters += estimate.numel() + readout_probe, readout_response = observations["readout"] + estimate_readout = -(readout_probe.t() @ readout_response) / ( + variance * readout_probe.shape[0]) + if estimate_readout.shape != net.R_out.shape: + raise ValueError("mirror readout response shape mismatch") + update_readout = estimate_readout - net.R_out + net.R_out.add_(update_readout, alpha=eta) + update_power += float(update_readout.square().sum()) + estimate_power += float(estimate_readout.square().sum()) + parameters += estimate_readout.numel() + return { + "mirror_update_rms": math.sqrt(update_power / parameters), + "mirror_estimate_rms": math.sqrt(estimate_power / parameters), + "conv_batch_size": int(observations["conv_batch_size"]), + "readout_batch_size": int(observations["readout_batch_size"]), + }, {"Q": estimates, "R": estimate_readout} + + +@torch.no_grad() +def normalized_response_mirror_step( + net, batch_size=1, noise_std=1.0, eta=0.1, generator=None): + observations = hierarchical_mirror_observations( + net, batch_size=batch_size, noise_std=noise_std, + generator=generator) + metrics, _ = normalized_response_mirror_update( + net, observations, eta=eta) + return metrics + + def conv_hierarchical_step(net, x, y, config): """One hierarchical-FA update using no reverse-mode graph or weight transport.""" config.validate() |