diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-07-22 20:32:19 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-07-22 20:32:19 -0500 |
| commit | 32122d026176ed199271aea243ccb0eef78eb30f (patch) | |
| tree | aaa78f5439625d7f39663a38275eacc447d85eb4 | |
| parent | 9ca51a97119efc90bc2a0b6b65624dc87689e088 (diff) | |
protocol: freeze oral B recovery confirmation
| -rw-r--r-- | ORAL_B_RECOVERY.md | 58 | ||||
| -rw-r--r-- | ROADMAP.md | 7 | ||||
| -rwxr-xr-x | experiments/analyze_bci_td_confirmation.py | 374 | ||||
| -rwxr-xr-x | experiments/bci_td_confirmation.py | 199 | ||||
| -rwxr-xr-x | experiments/bci_td_confirmation.sh | 16 | ||||
| -rwxr-xr-x | experiments/finalize_accept.sh | 4 |
6 files changed, 657 insertions, 1 deletions
diff --git a/ORAL_B_RECOVERY.md b/ORAL_B_RECOVERY.md index 7ab6785..da018dd 100644 --- a/ORAL_B_RECOVERY.md +++ b/ORAL_B_RECOVERY.md @@ -118,3 +118,61 @@ necessity, innovation identification, and a causal plasticity lesion. The executable runner, complete-grid analyzer, and shell entry point are `experiments/bci_td_run.py`, `experiments/analyze_bci_td_development.py`, and `experiments/bci_td_development_screen.sh`. None may run before D4 closes. + +## R2 frozen untouched confirmation + +R2 is committed before any R1 task endpoint and is executable only if the +complete R1 gate selects one eligible rate. It runs that rate without further +selection over untouched task seeds 10--15 and model seeds 0--4: exactly 30 +paired records, with no deletion or replacement. Each record repeats intact, +fixed-vectorizer, plasticity-lesion, and exact-role diagnostic conditions on +identical trajectories. Its separate 256-episode evaluation uses task seed +`300000 + training_task_seed`. The R2 runner requires clean tracked code, +protocol, D4 gate, and R1 gate and binds all of them by SHA-256. + +The task seed, rather than each of the five models sharing its environment, is +the independent unit for uncertainty. Every aggregate confidence check first +averages model seeds within each of the six task seeds and then uses a +one-sided 95% Student-t bound with five degrees of freedom. R2 requires all of +the following learning/plasticity checks: + +- mean intact final success at least 70%, every task mean at least 60%, and + the lower confidence bound at least 60%; +- mean early-to-late learning gain at least 10 points and its lower bound at + least 5 points; +- mean paired final gap over fixed vectorizer at least 20 points and its lower + bound at least 10 points; +- mean deficit to the exact-role diagnostic at most 10 points and its upper + bound at most 20 points; +- the lower bound of `0.5 * intact_gain - plasticity_lesion_gain` is + nonnegative; and +- mean learned-role cosine at least 0.80 and every one of 30 cosines at least + 0.70. + +The biological checks retain the thresholds frozen for the original B1/B2 +programme and add only task-cluster robustness bounds: + +- mean absolute innovation-soma correlation at most 0.10 (upper bound 0.12), + with mean raw-minus-innovation correlation at least 0.20 (lower bound 0.15); +- preceding surrounding-network state predicts amplification at mean balanced + accuracy at least 55% (lower bound 52%), while decoder distance has mean + residual correlation at least 0.10 (lower bound 0.02); +- residual population outcome accuracy averages at least 57% (lower bound + 53%) and exceeds soma by at least 3 points on average with nonnegative lower + bound; +- causal-role sign inversion is positive in at least 25/30 records and in all + six task-cluster means; +- performance velocity has at least 0.05 mean absolute CV-correlation + advantage over instantaneous error, with nonnegative lower bound; and +- early residual predicts late-minus-early somatic activity with mean + correlation at least 0.30 and lower bound at least 0.10. + +The executable runner, immutable complete-grid analyzer, and entry point are +`experiments/bci_td_confirmation.py`, +`experiments/analyze_bci_td_confirmation.py`, and +`experiments/bci_td_confirmation.sh`. The analyzer accepts exactly the 30 +named records. A complete pass raises the strict reviewer score from 7 to 8 +and establishes oral-B **innovation-guided plasticity**. Because `kappa=0`, it +explicitly does not revive the failed online-control or desired-velocity +claim. Any failed R2 gate is retained, leaves the score at 7, and closes this +recovery without threshold repair. @@ -44,6 +44,13 @@ seed, or confirmation seed is touched. The two-rate, three-task-seed R1 learning protocol and complete-grid analyzer are now committed, but its runner is hard-gated on a complete D4 pass. Confirmation seeds remain sealed. +The untouched recovery confirmation is now also frozen before any R1 task +endpoint. If and only if R1 selects an eligible rate, R2 runs task seeds +10--15 by model seeds 0--4 with task-cluster uncertainty, the original B1/B2 +signature thresholds, and a plasticity lesion. A complete R2 pass supports +innovation-guided plasticity and raises the strict score from 7 to 8; it cannot +support online control because the recovery fixes `kappa=0`. + ## Frozen accept claims and gates ### C1. Innovation is necessary under naturally mixed apical traffic diff --git a/experiments/analyze_bci_td_confirmation.py b/experiments/analyze_bci_td_confirmation.py new file mode 100755 index 0000000..8697e38 --- /dev/null +++ b/experiments/analyze_bci_td_confirmation.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +"""Audit the frozen oral-B temporal-difference confirmation panel.""" +import argparse +import glob +import hashlib +import json +import math +import os +import statistics + + +TASK_SEEDS = tuple(range(10, 16)) +MODEL_SEEDS = tuple(range(5)) +CONDITIONS = ("intact", "fixed_vectorizer", "plasticity_lesion", "oracle_role") +T_CRITICAL_ONE_SIDED_95_DF5 = 2.015048373 +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PROTOCOL_PATH = os.path.join(ROOT, "ORAL_B_RECOVERY.md") +RUNNER_PATH = os.path.join(ROOT, "experiments", "bci_td_confirmation.py") +DEVELOPMENT_RUNNER_PATH = os.path.join(ROOT, "experiments", "bci_td_run.py") + + +def require(condition, message): + if not condition: + raise ValueError(message) + + +def sha256(path): + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def finite_tree(value): + if isinstance(value, dict): + return all(finite_tree(item) for item in value.values()) + if isinstance(value, (list, tuple)): + return all(finite_tree(item) for item in value) + if isinstance(value, (int, float)): + return math.isfinite(value) + return True + + +def lower_confidence_bound(values): + return (statistics.mean(values) + - T_CRITICAL_ONE_SIDED_95_DF5 + * statistics.stdev(values) / math.sqrt(len(values))) + + +def upper_confidence_bound(values): + return (statistics.mean(values) + + T_CRITICAL_ONE_SIDED_95_DF5 + * statistics.stdev(values) / math.sqrt(len(values))) + + +def task_cluster_values(records, getter): + return [statistics.mean( + getter(records[(task_seed, model_seed)]) + for model_seed in MODEL_SEEDS) + for task_seed in TASK_SEEDS] + + +def summarize(values): + return { + "by_task_seed": values, + "mean": statistics.mean(values), + "one_sided_95pct_lower": lower_confidence_bound(values), + "one_sided_95pct_upper": upper_confidence_bound(values), + } + + +def validate(row, path, eta, digests): + require(row.get("schema_version") == 1, f"{path}: schema") + args = row.get("args", {}) + task_seed = args.get("task_seed") + model_seed = args.get("model_seed") + require(task_seed in TASK_SEEDS, f"{path}: task seed") + require(model_seed in MODEL_SEEDS, f"{path}: model seed") + protocol = row.get("protocol", {}) + require(protocol.get("name") == "oral_b_td_confirmation_v1", + f"{path}: protocol") + require(protocol.get("split") == "untouched_confirmation", + f"{path}: split") + require(protocol.get("training_task_seed") == task_seed, + f"{path}: training seed") + require(protocol.get("evaluation_task_seed") == task_seed + 300_000, + f"{path}: evaluation seed") + require(protocol.get("selected_eta") == eta, f"{path}: selected eta") + require(protocol.get("no_further_selection") is True, + f"{path}: post-selection tuning") + require(protocol.get("confirmation_grid_size") == 30, + f"{path}: grid size") + for key in ("d4_gate", "r1_gate", "protocol"): + require(protocol.get(f"{key}_sha256") == digests[key], + f"{path}: {key} digest") + + source = row.get("provenance", {}) + require(source.get("git_tracked_dirty") is False, f"{path}: dirty") + require(all(source.get("tracked_inputs", {}).values()) + and set(source.get("tracked_inputs", {})) == { + "runner", "development_runner", "protocol", "d4_gate", "r1_gate"}, + f"{path}: untracked input") + expected_source_digests = { + "runner": digests["runner"], + "development_runner": digests["development_runner"], + "protocol": digests["protocol"], + "d4_gate": digests["d4_gate"], + "r1_gate": digests["r1_gate"], + } + require(source.get("input_sha256") == expected_source_digests, + f"{path}: source digest drift") + require(isinstance(source.get("git_commit"), str) + and len(source["git_commit"]) == 40, f"{path}: commit") + + config = row.get("config", {}) + expected = { + "n_plus": 5, "n_minus": 5, "n_background": 30, + "context_dim": 16, "steps_per_episode": 28, + "episodes_per_day": 64, "days": 14, "target": 0.8, + "inertia": 0.65, "process_noise": 0.12, "context_ar": 0.8, + "coupling_scale": 1.0, "predictor_eta": 0.2, + "vectorizer_eta": 0.03, "forward_eta": eta, + "perturb_sigma": 0.03, "perturb_every": 4, + "kappa": 0.0, "feedback": "performance_velocity", + } + for key, value in expected.items(): + require(config.get(key) == value, f"{path}: config {key}") + + require(row.get("finite") is True and finite_tree(row), f"{path}: finite") + for name in CONDITIONS: + warmup = row["warmup"][name] + require(warmup["batches"] == 100 and warmup["examples"] == 6400, + f"{path}: warmup count {name}") + require(warmup["instruction_present"] is False, + f"{path}: warmup instruction {name}") + require(warmup["role_cursor_scalar_observations"] == 12800, + f"{path}: warmup observations {name}") + require(warmup["predictor_max_abs_error"] <= 1e-5, + f"{path}: predictor {name}") + condition = row["conditions"][name] + require(len(condition["daily_success"]) == 14, + f"{path}: trajectory {name}") + cost = condition["cost"] + require(cost["ordinary_state_episode_steps"] == 25088, + f"{path}: ordinary cost {name}") + require(cost["online_role_perturbation_events"] == 98, + f"{path}: event cost {name}") + expected_online = 12544 if name in ( + "intact", "plasticity_lesion") else 0 + require(cost["conservative_online_cursor_scalar_observations"] + == expected_online, f"{path}: cursor cost {name}") + require(row["signatures"]["evaluation_episodes"] == 256, + f"{path}: evaluation episodes") + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--results", default="results/bci_td_confirmation") + parser.add_argument( + "--d4_gate", + default="results/kp_dynamic_projection_confirmation_gate.json") + parser.add_argument("--r1_gate", default="results/bci_td_dev_gate.json") + parser.add_argument("--out", default="results/bci_td_confirmation_gate.json") + args = parser.parse_args() + with open(args.d4_gate) as handle: + d4 = json.load(handle) + with open(args.r1_gate) as handle: + r1 = json.load(handle) + require(d4.get("protocol") == + "kp_dynamic_neutral_projection_confirmation_v1" + and d4.get("status") == "passed" + and d4.get("review_score_after") == 7, "D4 gate") + require(r1.get("protocol") == "oral_b_td_development_v1" + and r1.get("status") == "passed" + and r1.get("complete_grid") is True + and r1.get("oral_b_confirmation_opened") is True + and r1.get("confirmation_seeds_touched") is False, + "R1 gate") + eta = float(r1["selected"]["eta"]) + require(eta in (0.03, 0.1), "R1 selected eta") + digests = { + "runner": sha256(RUNNER_PATH), + "development_runner": sha256(DEVELOPMENT_RUNNER_PATH), + "protocol": sha256(PROTOCOL_PATH), + "d4_gate": sha256(args.d4_gate), + "r1_gate": sha256(args.r1_gate), + } + require(r1.get("d4_gate_sha256") == digests["d4_gate"], + "R1/D4 digest binding") + + expected_names = { + f"bci_td_confirm_t{task_seed}_m{model_seed}.json" + for task_seed in TASK_SEEDS for model_seed in MODEL_SEEDS + } + observed_names = { + os.path.basename(path) + for path in glob.glob(os.path.join(args.results, "*.json")) + } + require(observed_names == expected_names, + f"confirmation grid drift: missing={sorted(expected_names-observed_names)}, " + f"extra={sorted(observed_names-expected_names)}") + records = {} + commits = set() + source_sha256 = {} + for task_seed in TASK_SEEDS: + for model_seed in MODEL_SEEDS: + path = os.path.join( + args.results, + f"bci_td_confirm_t{task_seed}_m{model_seed}.json") + with open(path) as handle: + row = json.load(handle) + validate(row, path, eta, digests) + records[(task_seed, model_seed)] = row + commits.add(row["provenance"]["git_commit"]) + source_sha256[path] = sha256(path) + require(len(commits) == 1, "R2 source revision drift") + + metrics = {} + getters = { + "intact_final": lambda r: r["conditions"]["intact"]["final_success"], + "intact_gain": lambda r: r["conditions"]["intact"]["learning_gain"], + "fixed_final_gap": lambda r: ( + r["conditions"]["intact"]["final_success"] + - r["conditions"]["fixed_vectorizer"]["final_success"]), + "oracle_final_deficit": lambda r: ( + r["conditions"]["oracle_role"]["final_success"] + - r["conditions"]["intact"]["final_success"]), + "plasticity_half_margin": lambda r: ( + 0.5 * r["conditions"]["intact"]["learning_gain"] + - r["conditions"]["plasticity_lesion"]["learning_gain"]), + "role_cosine": lambda r: r["conditions"]["intact"][ + "role_cosine_after_training"], + "residual_soma_corr": lambda r: r["signatures"][ + "mean_abs_residual_soma_corr"], + "raw_residual_corr_gap": lambda r: r["signatures"][ + "raw_minus_residual_abs_soma_corr"], + "surrounding_event_accuracy": lambda r: r["signatures"][ + "surrounding_event_decoder_balanced_acc"], + "decoder_distance_corr": lambda r: r["signatures"][ + "decoder_distance_residual_corr"], + "residual_outcome_accuracy": lambda r: r["signatures"][ + "residual_outcome_decoder_balanced_acc"], + "residual_soma_outcome_gap": lambda r: r["signatures"][ + "residual_minus_soma_outcome_acc"], + "sign_inversion": lambda r: r["signatures"][ + "causal_role_sign_inversion_index"], + "velocity_advantage": lambda r: r["signatures"][ + "velocity_minus_error_abs_cv_corr"], + "longitudinal_prediction": lambda r: r["signatures"][ + "early_residual_late_activity_change_corr"], + } + clustered = {} + for name, getter in getters.items(): + values = task_cluster_values(records, getter) + clustered[name] = values + metrics[name] = summarize(values) + + all_role_cosines = [getters["role_cosine"](row) + for row in records.values()] + all_signs = [getters["sign_inversion"](row) + for row in records.values()] + learning_checks = { + "mean_intact_final_at_least_0p70": + metrics["intact_final"]["mean"] >= 0.70, + "every_task_mean_final_at_least_0p60": + min(clustered["intact_final"]) >= 0.60, + "intact_final_lower_bound_at_least_0p60": + metrics["intact_final"]["one_sided_95pct_lower"] >= 0.60, + "mean_learning_gain_at_least_0p10": + metrics["intact_gain"]["mean"] >= 0.10, + "learning_gain_lower_bound_at_least_0p05": + metrics["intact_gain"]["one_sided_95pct_lower"] >= 0.05, + "mean_fixed_gap_at_least_0p20": + metrics["fixed_final_gap"]["mean"] >= 0.20, + "fixed_gap_lower_bound_at_least_0p10": + metrics["fixed_final_gap"]["one_sided_95pct_lower"] >= 0.10, + "mean_oracle_deficit_at_most_0p10": + metrics["oracle_final_deficit"]["mean"] <= 0.10, + "oracle_deficit_upper_bound_at_most_0p20": + metrics["oracle_final_deficit"]["one_sided_95pct_upper"] <= 0.20, + "plasticity_lesion_half_margin_lower_bound_nonnegative": + metrics["plasticity_half_margin"]["one_sided_95pct_lower"] >= 0.0, + "mean_role_cosine_at_least_0p80": + metrics["role_cosine"]["mean"] >= 0.80, + "every_role_cosine_at_least_0p70": min(all_role_cosines) >= 0.70, + } + innovation_checks = { + "mean_residual_soma_corr_at_most_0p10": + metrics["residual_soma_corr"]["mean"] <= 0.10, + "residual_soma_corr_upper_bound_at_most_0p12": + metrics["residual_soma_corr"]["one_sided_95pct_upper"] <= 0.12, + "mean_raw_residual_corr_gap_at_least_0p20": + metrics["raw_residual_corr_gap"]["mean"] >= 0.20, + "raw_residual_corr_gap_lower_bound_at_least_0p15": + metrics["raw_residual_corr_gap"]["one_sided_95pct_lower"] >= 0.15, + "mean_surrounding_event_accuracy_at_least_0p55": + metrics["surrounding_event_accuracy"]["mean"] >= 0.55, + "surrounding_event_accuracy_lower_bound_at_least_0p52": + metrics["surrounding_event_accuracy"]["one_sided_95pct_lower"] >= 0.52, + "mean_decoder_distance_corr_at_least_0p10": + metrics["decoder_distance_corr"]["mean"] >= 0.10, + "decoder_distance_corr_lower_bound_at_least_0p02": + metrics["decoder_distance_corr"]["one_sided_95pct_lower"] >= 0.02, + } + vectorization_checks = { + "mean_residual_outcome_accuracy_at_least_0p57": + metrics["residual_outcome_accuracy"]["mean"] >= 0.57, + "residual_outcome_accuracy_lower_bound_at_least_0p53": + metrics["residual_outcome_accuracy"]["one_sided_95pct_lower"] >= 0.53, + "mean_residual_soma_outcome_gap_at_least_0p03": + metrics["residual_soma_outcome_gap"]["mean"] >= 0.03, + "residual_soma_outcome_gap_lower_bound_nonnegative": + metrics["residual_soma_outcome_gap"]["one_sided_95pct_lower"] >= 0.0, + "positive_sign_inversion_in_at_least_25_of_30": + sum(value > 0 for value in all_signs) >= 25, + "positive_sign_inversion_in_every_task_cluster": + min(clustered["sign_inversion"]) > 0.0, + "mean_velocity_advantage_at_least_0p05": + metrics["velocity_advantage"]["mean"] >= 0.05, + "velocity_advantage_lower_bound_nonnegative": + metrics["velocity_advantage"]["one_sided_95pct_lower"] >= 0.0, + "mean_longitudinal_prediction_at_least_0p30": + metrics["longitudinal_prediction"]["mean"] >= 0.30, + "longitudinal_prediction_lower_bound_at_least_0p10": + metrics["longitudinal_prediction"]["one_sided_95pct_lower"] >= 0.10, + } + checks = { + "all_records_finite_paired_and_cost_audited": True, + "learning_and_plasticity": learning_checks, + "innovation_identification_and_network_prediction": innovation_checks, + "outcome_and_causal_role_vectorization": vectorization_checks, + } + passed = all( + value + for category, values in checks.items() + for value in ([values] if isinstance(values, bool) else values.values())) + output = { + "protocol": "oral_b_td_confirmation_v1", + "status": "passed" if passed else "failed", + "complete_grid": True, + "selected_eta": eta, + "checks": checks, + "metrics": metrics, + "positive_sign_count": sum(value > 0 for value in all_signs), + "source_commit": next(iter(commits)), + "source_sha256": source_sha256, + "d4_gate_sha256": digests["d4_gate"], + "r1_gate_sha256": digests["r1_gate"], + "protocol_sha256": digests["protocol"], + "oral_b_plasticity_innovation_established": passed, + "online_control_or_desired_velocity_established": False, + "review_score_before": 7, + "review_score_after": 8 if passed else 7, + "score_change_rule": ( + "only a complete untouched R2 pass establishes oral-B " + "innovation-guided plasticity; kappa=0 cannot establish online control"), + } + os.makedirs(os.path.dirname(os.path.abspath(args.out)), exist_ok=True) + if os.path.exists(args.out): + with open(args.out) as handle: + existing = json.load(handle) + require(existing == output, + "existing R2 gate differs from deterministic re-audit") + print(json.dumps(output, indent=2)) + return + with open(args.out, "w") as handle: + json.dump(output, handle, indent=2, sort_keys=True) + handle.write("\n") + print(json.dumps(output, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/experiments/bci_td_confirmation.py b/experiments/bci_td_confirmation.py new file mode 100755 index 0000000..2a5590d --- /dev/null +++ b/experiments/bci_td_confirmation.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python3 +"""Run one cell of the frozen oral-B temporal-difference confirmation.""" +import argparse +import hashlib +import json +import math +import os +import platform +import resource +import subprocess +import sys +import time + +import torch + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +from experiments.bci_td_run import (CONDITIONS, build_config, evaluate, + neutral_warmup, train) +from sdil.bci import BCISDIL, generate_trajectories +from sdil.bci_metrics import signature_metrics + + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +PROTOCOL_PATH = os.path.join(ROOT, "ORAL_B_RECOVERY.md") +TASK_SEEDS = tuple(range(10, 16)) +MODEL_SEEDS = tuple(range(5)) + + +def sha256(path): + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def finite_tree(value): + if isinstance(value, dict): + return all(finite_tree(item) for item in value.values()) + if isinstance(value, (list, tuple)): + return all(finite_tree(item) for item in value) + if isinstance(value, (int, float)): + return math.isfinite(value) + return True + + +def provenance(d4_gate, r1_gate): + def run(command): + return subprocess.run( + command, cwd=ROOT, check=True, capture_output=True, + text=True).stdout.strip() + + paths = { + "runner": os.path.abspath(__file__), + "development_runner": os.path.join(ROOT, "experiments", "bci_td_run.py"), + "protocol": PROTOCOL_PATH, + "d4_gate": os.path.abspath(d4_gate), + "r1_gate": os.path.abspath(r1_gate), + } + relative = {name: os.path.relpath(path, ROOT) + for name, path in paths.items()} + tracked = { + name: subprocess.run( + ["git", "ls-files", "--error-unmatch", path], cwd=ROOT, + capture_output=True).returncode == 0 + for name, path in relative.items() + } + return { + "git_commit": run(["git", "rev-parse", "HEAD"]), + "git_tracked_dirty": bool(run( + ["git", "status", "--porcelain", "--untracked-files=no"])), + "tracked_inputs": tracked, + "input_sha256": {name: sha256(path) for name, path in paths.items()}, + } + + +def require_gate(d4, r1, d4_digest): + if not (d4.get("protocol") == + "kp_dynamic_neutral_projection_confirmation_v1" + and d4.get("status") == "passed" + and d4.get("review_score_after") == 7): + raise ValueError("oral-B R2 requires the complete audited D4 pass") + if not (r1.get("protocol") == "oral_b_td_development_v1" + and r1.get("status") == "passed" + and r1.get("complete_grid") is True + and r1.get("confirmation_seeds_touched") is False + and r1.get("oral_b_confirmation_opened") is True + and r1.get("review_score_after") == 7 + and r1.get("d4_gate_sha256") == d4_digest + and r1.get("selected", {}).get("eta") in (0.03, 0.1)): + raise ValueError("oral-B R2 requires the complete eligible R1 gate") + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--d4_gate", + default="results/kp_dynamic_projection_confirmation_gate.json") + parser.add_argument( + "--r1_gate", default="results/bci_td_dev_gate.json") + parser.add_argument("--task-seed", type=int, choices=TASK_SEEDS, required=True) + parser.add_argument("--model-seed", type=int, choices=MODEL_SEEDS, required=True) + parser.add_argument("--outdir", default="results/bci_td_confirmation") + args = parser.parse_args() + + with open(args.d4_gate) as handle: + d4 = json.load(handle) + with open(args.r1_gate) as handle: + r1 = json.load(handle) + d4_digest = sha256(args.d4_gate) + require_gate(d4, r1, d4_digest) + eta = float(r1["selected"]["eta"]) + source = provenance(args.d4_gate, args.r1_gate) + if (source["git_tracked_dirty"] + or not all(source["tracked_inputs"].values())): + raise RuntimeError( + "R2 requires clean tracked code, protocol, D4 gate, and R1 gate") + + cfg = build_config(eta) + trajectories = generate_trajectories(cfg, args.task_seed) + evaluation_seed = args.task_seed + 300_000 + evaluation = generate_trajectories( + cfg, evaluation_seed, days=1, episodes=256) + initial = BCISDIL(cfg, args.model_seed) + conditions = {} + trained = {} + warmups = {} + training_events = None + started = time.perf_counter() + for name in CONDITIONS: + model, warmup = neutral_warmup( + initial, args.task_seed, args.model_seed, name) + events, report = train( + model, trajectories, name, collect=name == "intact") + warmups[name] = warmup + trained[name] = model + conditions[name] = report + if name == "intact": + training_events = events + + evaluation_events = None + for name in CONDITIONS: + report = evaluate( + trained[name], evaluation, collect=name == "intact") + conditions[name]["final_success"] = report["success_rate"] + if name == "intact": + evaluation_events = report["events"] + signatures = signature_metrics( + training_events, evaluation_events, cfg, trained["intact"].role) + + result = { + "schema_version": 1, + "protocol": { + "name": "oral_b_td_confirmation_v1", + "split": "untouched_confirmation", + "training_task_seed": args.task_seed, + "evaluation_task_seed": evaluation_seed, + "selected_eta": eta, + "no_further_selection": True, + "confirmation_grid_size": len(TASK_SEEDS) * len(MODEL_SEEDS), + "d4_gate_sha256": source["input_sha256"]["d4_gate"], + "r1_gate_sha256": source["input_sha256"]["r1_gate"], + "protocol_sha256": source["input_sha256"]["protocol"], + }, + "args": vars(args), + "config": vars(cfg), + "provenance": source, + "warmup": warmups, + "conditions": conditions, + "signatures": signatures, + "wall_s": time.perf_counter() - started, + "peak_rss_mib": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, + "hardware": { + "device": "cpu", "platform": platform.platform(), + "torch_version": torch.__version__, "threads": torch.get_num_threads(), + }, + } + result["finite"] = finite_tree(result) + if not result["finite"]: + raise RuntimeError("non-finite oral-B R2 record") + os.makedirs(args.outdir, exist_ok=True) + path = os.path.join( + args.outdir, + f"bci_td_confirm_t{args.task_seed}_m{args.model_seed}.json") + if os.path.exists(path): + raise FileExistsError(f"refusing to overwrite {path}") + with open(path, "w") as handle: + json.dump(result, handle, indent=2, sort_keys=True) + handle.write("\n") + print(json.dumps({ + "path": path, + "finite": result["finite"], + "intact_final": conditions["intact"]["final_success"], + "sign_inversion": signatures["causal_role_sign_inversion_index"], + }, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/experiments/bci_td_confirmation.sh b/experiments/bci_td_confirmation.sh new file mode 100755 index 0000000..aab8c8c --- /dev/null +++ b/experiments/bci_td_confirmation.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash +# Frozen R1-gated oral-B confirmation: 6 task seeds x 5 model seeds. +set -eu + +cd "$(dirname "$0")/.." +PY=/home/yurenh2/miniconda3/envs/ep_pascal/bin/python3 + +for task_seed in 10 11 12 13 14 15; do + for model_seed in 0 1 2 3 4; do + OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 "$PY" \ + experiments/bci_td_confirmation.py \ + --task-seed "$task_seed" --model-seed "$model_seed" + done +done + +"$PY" experiments/analyze_bci_td_confirmation.py diff --git a/experiments/finalize_accept.sh b/experiments/finalize_accept.sh index d727f2e..d2aa4cc 100755 --- a/experiments/finalize_accept.sh +++ b/experiments/finalize_accept.sh @@ -13,7 +13,9 @@ experiments/finalize_claims.sh /home/yurenh2/miniconda3/envs/ep_pascal/bin/python3 \ experiments/bci_smoke.py /home/yurenh2/miniconda3/envs/ep_pascal/bin/python3 -m py_compile \ - experiments/bci_td_run.py experiments/analyze_bci_td_development.py + experiments/bci_td_run.py experiments/analyze_bci_td_development.py \ + experiments/bci_td_confirmation.py \ + experiments/analyze_bci_td_confirmation.py /home/yurenh2/miniconda3/envs/ep_pascal/bin/python3 \ experiments/analyze_kp_dynamic_projection.py >/dev/null /home/yurenh2/miniconda3/envs/ep_pascal/bin/python3 \ |
