summaryrefslogtreecommitdiff
path: root/docs/metrics.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/metrics.md')
-rw-r--r--docs/metrics.md60
1 files changed, 60 insertions, 0 deletions
diff --git a/docs/metrics.md b/docs/metrics.md
index e69de29..553a4af 100644
--- a/docs/metrics.md
+++ b/docs/metrics.md
@@ -0,0 +1,60 @@
+# Metrics & Notation (EN-only, T = 0)
+
+## Notation
+- Vocabulary: \(\mathcal V\). Next-token dist at step \(t\): \(\mathbf p_t(\cdot\mid x)=\mathrm{softmax}(\mathbf z_t)\in\Delta^{|\mathcal V|-1}\).
+- Female/Male token sets: \(F, M\subset \mathcal V\)(由 `assets/groups/en_*` 映射得到)。
+- Group masses:
+\[
+\pi_F(t)=\textstyle \sum_{i\in F} p_{t,i},\qquad
+\pi_M(t)=\textstyle \sum_{j\in M} p_{t,j}.
+\]
+- Gating mask per step: \(w_t\in\{0,1\}\)(见 `docs/gating.md`)。
+- Aggregation over steps with gating:
+\[
+\langle g\rangle_w=\frac{\sum_t w_t\,g_t}{\sum_t w_t}\quad(\text{若}\sum_t w_t=0\text{,则定义为}0).
+\]
+
+---
+
+## Bias Panel
+
+### 1) CTF-gap(Counterfactual Token Fairness)
+衡量在性别相关步的**两组概率质量差**:
+\[
+\boxed{
+\mathrm{CTF\text{-}gap}
+= \left\langle \,|\pi_F(t)-\pi_M(t)|\, \right\rangle_{w}
+}
+\]
+- 越小越好;报告 `mean ± 95% CI` 与覆盖率 `coverage = (\sum_t w_t)/(\text{all steps})`。
+
+### 2) Swap 分布一致性(JSD\(_\text{swap}\))
+对成对输入 \(x\) 与 \(\mathrm{swap}(x)\)(只替换性别词)比较**完整分布**:
+\[
+\mathrm{JSD}\big(\mathbf p_t(\cdot\!\mid\!x),\mathbf p_t(\cdot\!\mid\!\mathrm{swap}(x))\big)
+=\tfrac12\mathrm{KL}\!\big(\mathbf p_t\| \mathbf m_t\big)
++\tfrac12\mathrm{KL}\!\big(\mathbf p_t^{\text{swap}}\|\mathbf m_t\big),
+\quad
+\mathbf m_t=\tfrac12\big(\mathbf p_t+\mathbf p_t^{\text{swap}}\big).
+\]
+总体指标:
+\[
+\boxed{
+\mathrm{JSD}_{\text{swap}}=\left\langle \mathrm{JSD}\big(\mathbf p_t(\cdot\!\mid\!x),\mathbf p_t(\cdot\!\mid\!\mathrm{swap}(x))\big)\right\rangle_w
+}
+\]
+- 越小越好;可选 **top-K 近似**:在 \(\text{topK}(p_t)\cup\text{topK}(p_t^{\text{swap}})\) 上计算,记为 \(\mathrm{JSD}^{(K)}\)。
+
+### 3) Swap 答案一致性(AnsConsistency)
+\[
+\boxed{
+\mathrm{AnsCons}=\frac{1}{N}\sum_{n=1}^{N}\mathbf 1\big\{\hat y(x_n)=\hat y(\mathrm{swap}(x_n))\big\}
+}
+\]
+- \(\hat y(\cdot)\) 为统一解码(本项目 \(T=0\))下的输出或选择。越大越好。
+
+### 4) CrowS-Pairs(gender)
+对数似然优势(anti - stereo):
+\[
+\boxed{
+\Delta\log P=\log P(\