diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-08-10 10:25:00 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-08-10 10:25:00 -0500 |
| commit | 5ab644db6823491ffadb793162be36f4aa5edd04 (patch) | |
| tree | ba70c4ea359b5a7735f7970a5d23d28e4b0e3779 /BABYAI_SHARED_FEEDBACK.md | |
| parent | e3011f144a018f6415b3c2a636c0646e412b3ced (diff) | |
results: retain GoToObj as diagnostic only
Diffstat (limited to 'BABYAI_SHARED_FEEDBACK.md')
| -rw-r--r-- | BABYAI_SHARED_FEEDBACK.md | 19 |
1 files changed, 19 insertions, 0 deletions
diff --git a/BABYAI_SHARED_FEEDBACK.md b/BABYAI_SHARED_FEEDBACK.md index d00af72..74188ea 100644 --- a/BABYAI_SHARED_FEEDBACK.md +++ b/BABYAI_SHARED_FEEDBACK.md @@ -126,3 +126,22 @@ frozen tie rule: its BP mission lesion reduced rollout success from `97.8%` to used the fewest layers and smaller learning rate. The selector read no raw, SDIL, or test result. This outcome opens B1 without establishing an SDIL advantage. + +## B1 outcome: diagnostic only + +The task-structure audit found exactly one ball, box, or key in each of 256 +generated `GoToObjS6` episodes. That sole object is always the target, so the +mission does not select among alternatives. The original claim that language +is required by this task was therefore false; mission lesion is only an input +ablation in this environment. + +The three-seed downstream result is nevertheless useful as an implementation +diagnostic. Mean rollout success was `97.8%` for BP, `97.8%` for clean KP, +`83.93%` for raw shared KP, and `95.87%` for SDIL. The paired SDIL-minus-raw +rollout differences were `+30.6`, `+5.6`, and `-0.4` points; expert-action +differences were `+12.22`, `+0.16`, and `-1.11` points. Thus mixed traffic can +damage learning and SDIL often recovers it, but the effect is not consistent +across seeds and the task does not satisfy the central endogenous-context +premise. B1 fails as paper evidence and is retained without a norm-matched +follow-up. The next BabyAI task is `PickupLoc`, which contains eight candidate +objects per audited episode. |
