1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
|
.. _probe_cards:
Probe Cards
===========
Each `Probe` is documented with its own ``README`` file. This document describes
the schema used to create these files. We describe the documents field by field
as they are written.
- Abstract
Abstract succintly describes the main idea behind the probe.
- Harms
Description of harms measured by the probe.
- Use case
What is the use case for using LLMs in the context of the prompt.
- Genders
What genders are considered.
- Genders definition
How is the gender indicated in the texts (explicitly stated, gender-coded
pronouns, gender-coded names, etc).
- Genders placement
Whose gender is being processed, e.g., author of a text, user, subject of
a text.
- Language
Natural language used in the prompts / responses.
- Output type
What is type of the output, e.g., structured responses, free text.
- Modality
What is the modality of the conversation, e.g., single turn text
chats, tools, image generation.
- Domain
What is domain of the data used, e.g., everyday life, healthcare, business.
- Realistic format
Is the format of prompts realistic? Is it possible that similar requests
could be used by common users? Do the queries make practical sense outside
of the probing context?
- Data source
How were the data created, e.g., human annotators, LLMs, scraping.
- Size
Number of probe items.
- Intersectionality
Are there non-gender-related harms that could be addressed by the probe,
e.g., race, occupation.
- Folder
Where is the code located.
- Methodology
- Probe Items
Description of how are the probe items created.
- Data
Description of the necessary data used to create the probe items.
- Evaluation
Description of the answer evaluation methodology.
- Metrics
Description of all the calculated metrics.
- Sources
List of all the resources that can improve the understanding of the probe,
e.g., related papers or datasets.
- Probe parameters
Documentation for the parameters used when the probe is initialized in the
code.
- Limitations / Improvements
Discussion about the limitations of the probe and ideas about how to improve
it in the future.
|
