diff options
Diffstat (limited to 'results/threshold_sensitivity_output.txt')
| -rw-r--r-- | results/threshold_sensitivity_output.txt | 105 |
1 files changed, 105 insertions, 0 deletions
diff --git a/results/threshold_sensitivity_output.txt b/results/threshold_sensitivity_output.txt new file mode 100644 index 0000000..083176b --- /dev/null +++ b/results/threshold_sensitivity_output.txt @@ -0,0 +1,105 @@ +======================================================================================== +DIAGNOSTIC (a) per-block growth: sensitivity over threshold +======================================================================================== +method seed value >5× >10× >20× >50× >100× >500× >1000× >5000× +bp 42 1.00e+00 ok ok ok ok ok ok ok ok +dfa 42 2.04e+03 FIRE FIRE FIRE FIRE FIRE FIRE FIRE ok +state_bridge 42 1.28e+04 FIRE FIRE FIRE FIRE FIRE FIRE FIRE FIRE +credit_bridge 42 1.82e+03 FIRE FIRE FIRE FIRE FIRE FIRE FIRE ok +ep 42 2.87e+00 ok ok ok ok ok ok ok ok +bp 123 9.59e-01 ok ok ok ok ok ok ok ok +dfa 123 9.78e+02 FIRE FIRE FIRE FIRE FIRE FIRE ok ok +state_bridge 123 2.41e+04 FIRE FIRE FIRE FIRE FIRE FIRE FIRE FIRE +credit_bridge 123 6.94e+02 FIRE FIRE FIRE FIRE FIRE FIRE ok ok +ep 123 1.10e+01 FIRE FIRE ok ok ok ok ok ok +bp 456 9.63e-01 ok ok ok ok ok ok ok ok +dfa 456 2.55e+03 FIRE FIRE FIRE FIRE FIRE FIRE FIRE ok +state_bridge 456 1.05e+04 FIRE FIRE FIRE FIRE FIRE FIRE FIRE FIRE +credit_bridge 456 1.03e+03 FIRE FIRE FIRE FIRE FIRE FIRE FIRE ok +ep 456 6.10e+00 FIRE ok ok ok ok ok ok ok + +Reading: BP/EP rows should be 'ok' across the entire row (the whole +threshold range is healthy for them). DFA/SB/CB rows should be 'FIRE' +at the chosen threshold and have a comfortable margin on either side. + +======================================================================================== +DIAGNOSTIC (b) g-norm floor: sensitivity over threshold +======================================================================================== +method seed value <1e-09 <1e-08 <1e-07 <1e-06 <1e-05 +bp 42 3.70e-04 ok ok ok ok ok +dfa 42 4.17e-09 ok FIRE FIRE FIRE FIRE +state_bridge 42 1.84e-09 ok FIRE FIRE FIRE FIRE +credit_bridge 42 9.01e-10 FIRE FIRE FIRE FIRE FIRE +ep 42 1.64e-04 ok ok ok ok ok +bp 123 3.09e-04 ok ok ok ok ok +dfa 123 2.84e-09 ok FIRE FIRE FIRE FIRE +state_bridge 123 2.24e-09 ok FIRE FIRE FIRE FIRE +credit_bridge 123 4.18e-09 ok FIRE FIRE FIRE FIRE +ep 123 1.02e-04 ok ok ok ok ok +bp 456 4.02e-04 ok ok ok ok ok +dfa 456 1.90e-09 ok FIRE FIRE FIRE FIRE +state_bridge 456 2.40e-09 ok FIRE FIRE FIRE FIRE +credit_bridge 456 2.38e-09 ok FIRE FIRE FIRE FIRE +ep 456 1.16e-04 ok ok ok ok ok + +======================================================================================== +DIAGNOSTIC (c) stability ceiling: sensitivity over threshold +======================================================================================== +method seed value >0.10 >0.20 >0.30 >0.50 >0.70 >0.90 +bp 42 0.099 ok ok ok ok ok ok +dfa 42 0.047 ok ok ok ok ok ok +state_bridge 42 0.992 FIRE FIRE FIRE FIRE FIRE FIRE +credit_bridge 42 0.352 FIRE FIRE FIRE ok ok ok +ep 42 -0.036 ok ok ok ok ok ok +bp 123 0.087 ok ok ok ok ok ok +dfa 123 0.436 FIRE FIRE FIRE ok ok ok +state_bridge 123 0.561 FIRE FIRE FIRE FIRE ok ok +credit_bridge 123 0.250 FIRE FIRE ok ok ok ok +ep 123 0.120 FIRE ok ok ok ok ok +bp 456 0.114 FIRE ok ok ok ok ok +dfa 456 -0.005 ok ok ok ok ok ok +state_bridge 456 0.035 ok ok ok ok ok ok +credit_bridge 456 0.518 FIRE FIRE FIRE FIRE ok ok +ep 456 -0.024 ok ok ok ok ok ok + +======================================================================================== +VERDICT ROBUSTNESS: at what threshold does each verdict CHANGE? +======================================================================================== +method seed (a) value (a) flip near (b) value (b) flip near +bp 42 1.00e+00 1.00e+00 3.70e-04 3.70e-04 +dfa 42 2.04e+03 2.04e+03 4.17e-09 4.17e-09 +state_bridge 42 1.28e+04 1.28e+04 1.84e-09 1.84e-09 +credit_bridge 42 1.82e+03 1.82e+03 9.01e-10 9.01e-10 +ep 42 2.87e+00 2.87e+00 1.64e-04 1.64e-04 +bp 123 9.59e-01 9.59e-01 3.09e-04 3.09e-04 +dfa 123 9.78e+02 9.78e+02 2.84e-09 2.84e-09 +state_bridge 123 2.41e+04 2.41e+04 2.24e-09 2.24e-09 +credit_bridge 123 6.94e+02 6.94e+02 4.18e-09 4.18e-09 +ep 123 1.10e+01 1.10e+01 1.02e-04 1.02e-04 +bp 456 9.63e-01 9.63e-01 4.02e-04 4.02e-04 +dfa 456 2.55e+03 2.55e+03 1.90e-09 1.90e-09 +state_bridge 456 1.05e+04 1.05e+04 2.40e-09 2.40e-09 +credit_bridge 456 1.03e+03 1.03e+03 2.38e-09 2.38e-09 +ep 456 6.10e+00 6.10e+00 1.16e-04 1.16e-04 + +Interpretation: + - For DFA/SB/CB, the 'flip near' values are the diagnostic raw values. + Default (a) threshold 50 catches all if raw values > 50; default (b) + threshold 1e-7 catches all if raw values < 1e-7. Compare: + (a) max BP per-block growth across 3 seeds: 1.00e+00 + (a) max EP per-block growth across 3 seeds: 1.10e+01 + (a) min DFA per-block growth across 3 seeds: 9.78e+02 + (a) min SB per-block growth across 3 seeds: 1.05e+04 + (a) min CB per-block growth across 3 seeds: 6.94e+02 + -> separation gap: healthy max = 1.10e+01, + degenerate min = 6.94e+02, + gap factor = 63× + + (b) min BP ‖g_L‖ across 3 seeds: 3.09e-04 + (b) min EP ‖g_L‖ across 3 seeds: 1.02e-04 + (b) max DFA ‖g_L‖ across 3 seeds: 4.17e-09 + (b) max SB ‖g_L‖ across 3 seeds: 2.40e-09 + (b) max CB ‖g_L‖ across 3 seeds: 4.18e-09 + -> separation gap: healthy min = 1.02e-04, + degenerate max = 4.18e-09, + gap factor = 24338× |