From 5ab644db6823491ffadb793162be36f4aa5edd04 Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Mon, 10 Aug 2026 10:25:00 -0500 Subject: results: retain GoToObj as diagnostic only --- BABYAI_SHARED_FEEDBACK.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) (limited to 'BABYAI_SHARED_FEEDBACK.md') diff --git a/BABYAI_SHARED_FEEDBACK.md b/BABYAI_SHARED_FEEDBACK.md index d00af72..74188ea 100644 --- a/BABYAI_SHARED_FEEDBACK.md +++ b/BABYAI_SHARED_FEEDBACK.md @@ -126,3 +126,22 @@ frozen tie rule: its BP mission lesion reduced rollout success from `97.8%` to used the fewest layers and smaller learning rate. The selector read no raw, SDIL, or test result. This outcome opens B1 without establishing an SDIL advantage. + +## B1 outcome: diagnostic only + +The task-structure audit found exactly one ball, box, or key in each of 256 +generated `GoToObjS6` episodes. That sole object is always the target, so the +mission does not select among alternatives. The original claim that language +is required by this task was therefore false; mission lesion is only an input +ablation in this environment. + +The three-seed downstream result is nevertheless useful as an implementation +diagnostic. Mean rollout success was `97.8%` for BP, `97.8%` for clean KP, +`83.93%` for raw shared KP, and `95.87%` for SDIL. The paired SDIL-minus-raw +rollout differences were `+30.6`, `+5.6`, and `-0.4` points; expert-action +differences were `+12.22`, `+0.16`, and `-1.11` points. Thus mixed traffic can +damage learning and SDIL often recovers it, but the effect is not consistent +across seeds and the task does not satisfy the central endogenous-context +premise. B1 fails as paper evidence and is retained without a norm-matched +follow-up. The next BabyAI task is `PickupLoc`, which contains eight candidate +objects per audited episode. -- cgit v1.2.3