From 6d99f411cf7e3f271c85976f27e3ad780f9ecbc8 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Mon, 10 Aug 2026 10:13:57 -0500 Subject: exp: freeze BabyAI shared-feedback protocol --- BABYAI_SHARED_FEEDBACK.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 BABYAI_SHARED_FEEDBACK.md (limited to 'BABYAI_SHARED_FEEDBACK.md') diff --git a/BABYAI_SHARED_FEEDBACK.md b/BABYAI_SHARED_FEEDBACK.md new file mode 100644 index 0000000..addd58a --- /dev/null +++ b/BABYAI_SHARED_FEEDBACK.md @@ -0,0 +1,116 @@ +# BabyAI shared-feedback protocol + +## Question + +The clean scaling experiments show that reciprocal Kolen--Pollack (KP) +feedback can train large networks, but do not show that SDIL adds anything to +KP. This experiment asks whether SDIL is useful when an ordinary top-down +signal is required during inference and the same physical pathway also carries +the teaching signal. + +BabyAI supplies this structure without an injected nuisance. The visual +observation is the basal input, while the language mission is required to +choose the correct object and enters each hidden population through a fixed +apical projection. During imitation learning, the measured apical signal is + +```text +a_l = C_l m + t_l, +``` + +where `C_l m` is the mission field used during inference and `t_l` is the +reciprocal KP teaching field. Raw shared-path learning uses `a_l`. SDIL fits +a per-cell affine prediction of `C_l m` from the local hidden activity and uses +`a_l - P_l(h_l)`. + +This is a fixed-distribution, randomly interleaved grounded-language task. It +is not a continual-learning experiment. The endpoint tests signal separation +and task learning, not catastrophic forgetting. + +## Data and task + +The first task is the official `BabyAI-GoToObjS6-v1` environment from +MiniGrid 3.1.0. Expert demonstrations are generated by the official +`BabyAIBot`. Every recorded trajectory must end with positive reward; failed +expert episodes abort data generation rather than being silently discarded. + +The basal observation is the standard partial `7 x 7 x 3` symbolic image plus +agent direction. Each of the object, color, and state channels is one-hot +encoded. The mission is a normalized bag of training-vocabulary tokens and is +not included in the basal input. Validation missions use the training +vocabulary, with an explicit unknown token retained for auditability. + +Development data contain 20,000 expert episodes for training and 2,000 +disjoint expert episodes for validation. A separate set of 500 validation +seeds is used for closed-loop policy rollouts. No test seeds are generated or +evaluated until the method configuration and mechanism comparison are frozen. + +## Model and local updates + +The policy is a tanh MLP with a fixed mission projection into every hidden +population and a seven-action linear readout: + +```text +u_l = W_l h_(l-1) + b_l + C_l m +h_l = tanh(u_l). +``` + +`C_l` is fixed after initialization. Forward weights and independently stored +reciprocal weights use the same modified-KP momentum, decay, and local +correlation rule used in the existing shared-feedback implementation. The +reciprocal tensor is updated from the same locally available activity product; +it is never copied from the forward tensor. SDIL predictor fitting receives +only the current hidden activity and ordinary mission field, never the expert +action, loss, output error, KP teaching field, or downstream weight. + +## B0 clean selector + +B0 exists only to choose a model that the task and clean learning rules can +train. It runs BP and clean KP, never raw shared KP or SDIL. The fixed grid is + +```text +hidden layers: 2, 4 +hidden width: 256 +learning rate: 0.01, 0.03 +context gain: 1.0 +epochs: 15 +batch size: 256 +model/data seed: 4101 +``` + +All candidates share demonstrations, minibatch order, initialization within a +model shape, and rollout seeds. A candidate is eligible only if BP and clean +KP each reach at least 80% validation rollout success and removing the mission +field reduces BP rollout success by at least 20 points. Among eligible +candidates, choose the highest clean-KP rollout success; ties are resolved by +action accuracy, then fewer layers, then the smaller learning rate. If no +candidate is eligible, B0 fails and the shared-path endpoint is not run. + +## B1 mechanism endpoint + +The selected depth and learning rate are frozen. B1 trains for 40 epochs on +three model seeds with identical data and minibatch orders across conditions: + +1. `bp`: exact backpropagation reference with the same fixed mission pathway. +2. `clean_kp`: instruction-only reciprocal KP; this receives a separate clean + teaching wire and is the local-learning upper bound. +3. `raw_shared`: the local teaching rule directly uses `C_l m + t_l`. +4. `sdil`: a per-cell neutral affine predictor is fitted on 4,096 training + observations before each epoch and subtracts the predicted mission field. + +The predictor's extra forward observations, wall time, and arithmetic are +reported. The main metrics are validation closed-loop mission success, +validation expert-action accuracy, mission-lesion success, cross-entropy, +predictor explained variance, and the residual mission-field RMS. Accuracy +and mission success, not residual size, decide the result. + +B1 supports the mechanism if clean KP is successful, mission removal damages +the task, raw shared KP is worse than clean KP, and SDIL recovers a substantial +part of that downstream success gap consistently across seeds. No fixed +five-point threshold is imposed before observing the natural effect size. +Norm-matched raw feedback is added only after a positive raw--SDIL difference, +because it diagnoses that difference but cannot create it. + +Passing B1 opens `PickupLoc` and `PutNextLocalS6N4` with the same method and +selection rule, followed by depth scaling. Failure is retained as evidence +that task-required shared feedback alone is insufficient to make +residualization useful in this setting. -- cgit v1.2.3