Reference

Documents

The document format field by field, with the JSON Schema.

A document is one JSON object and nothing else. It names a model, which column of a row each prompt comes from, the places inside the model to read and to write, the experiment those make up, and the steps that run it over rows. Nothing in it is code, and nothing in it is resolved: pos stays -1 and "weekdays/train" stays a string until the compiler resolves them against a loaded tokenizer and the rows on disk.

Two formats parse. The plan-shaped one below is the format — the one every document in documents/v2/ and documents/real/ is written in, and the one the schema verb describes. The other is a narrow reader for the protocol's own JSON, kept for the documents at the top of documents/. A document is told apart by shape: it is plan-shaped if it has steps.

At a glance #

The smallest real experiment in the repository: read the residual stream at the last token of a counterfactual prompt, swap it into the same place on the base prompt, score what the model then says.

documents/v2/patching.json — the whole document

{
  "header": {
    "description": "The smallest real interchange \u2014 one swap at the last position of layer 0 \u2014 in causalab-mini's plan-shaped format. Compare documents/minimal_cpu.json, the same experiment in the protocol's. Here there is one step, because the run is one pass."
  },
  "model": {
    "key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
    "revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
    "dtype": "fp32"
  },
  "roles": {
    "base": {
      "field": "input"
    },
    "counterfactual": {
      "field": "counterfactual_inputs[0]"
    }
  },
  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  },
  "interventions": {
    "patching": {
      "reads": {
        "v_cf": {
          "site": "target",
          "pos": -1,
          "model": "original",
          "input": "counterfactual"
        },
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "patched",
          "input": "base"
        }
      },
      "writes": {
        "patch": {
          "site": "target",
          "pos": -1,
          "mechanism": "swap",
          "operand": "v_cf"
        }
      },
      "models": {
        "patched": {
          "input": "base",
          "writes": [
            "patch"
          ]
        }
      },
      "metrics": {
        "iia": {
          "kind": "match",
          "of": "logits",
          "expected": "cf_answer",
          "token_form": "space_prefixed"
        },
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        }
      }
    }
  },
  "steps": {
    "score": {
      "kind": "observe",
      "interventions": "patching",
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "saves": [
        {
          "value": "iia",
          "file_path": "iia.json"
        },
        {
          "value": "logit_diff",
          "file_path": "logit_diff.json"
        }
      ]
    }
  }
}

$ uv run causalab-mini validate documents/v2/patching.json
ok: documents/v2/patching.json is a valid plan-shaped document

field	type	default	what it is
`header`	object	`{}`	authoring metadata. One key, `description`. Not part of the digest
`model`	object	required	the checkpoint, pinned to a revision and a dtype
`roles`	`{name: object}`	required	which column of a row each input role's text comes from
`sites`	`{name: object}`	required	the places a read or a write may name
`featurizers`	`{name: object}`	`{}`	parameter sets, by name
`interventions`	`{name: object}`	required	the experiments steps name: reads, writes, intervened models, metrics. One written inline on a step is declared here under the step's name, or `<step>[i]` as entry `i` of a list, and there must be at least one
`steps`	`{name: object}`	required	what runs, over which rows, and what comes out

every node refuses a key it does not know

Every object in the document is a pydantic model with extra="forbid", so a misspelling anywhere is a load error that names the path to it — interventions.patching.reads.v_cf.position — rather than a key that is quietly ignored and still changes the digest.

How a document fits together #

Everything above steps is a declaration. roles, sites, featurizers and interventions name things — a column, a place in the model, a parameter set, an experiment — and nothing there runs. A declared intervention no step names is never compiled.

A step is an execution: it binds rows to the interventions it runs, and every observe and fit says which with interventions. There is no default, even when the document declares only one: a step on its own tells you what it runs.

An intervention is declared once at the root, and not inside the step that runs it, because more than one thing refers to it. A fit and the observe that scores it name the same one, so the score measures exactly what was trained. A sweep rewrites a field of it once and every step that names it sees the new value. And a step's outputs and saves are named by what the intervention reads and scores, so two steps over different rows produce comparable results.

A baseline, then two experiments at one site run over the same rows by one step:

documents/v2/two_observes.json — the whole document

{
  "header": {
    "description": "Two experiments over the same rows, as one step, and a baseline written in place. `interventions` declares interchange patching and mean ablation at the last position of layer 0; nothing there runs. `clean` is an intervention with no writes, written inline on its step: it reads the original model, scores it, and publishes the mean activation at the site. `compare` lists both declared experiments, so each runs over its rows: `ablation` swaps in the mean `clean` published, `iia` belongs to patching alone and is saved by its bare name, and `logit_diff`, which both produce, is saved qualified. Tiny CPU Llama, so the numbers demonstrate the mechanism and nothing else."
  },
  "model": {
    "key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
    "revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
    "dtype": "fp32"
  },
  "roles": {
    "base": {
      "field": "input"
    },
    "counterfactual": {
      "field": "counterfactual_inputs[0]"
    }
  },
  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  },
  "interventions": {
    "patching": {
      "reads": {
        "v_cf": {
          "site": "target",
          "pos": -1,
          "model": "original",
          "input": "counterfactual"
        },
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "patched",
          "input": "base"
        }
      },
      "writes": {
        "patch": {
          "site": "target",
          "pos": -1,
          "mechanism": "swap",
          "operand": "v_cf"
        }
      },
      "models": {
        "patched": {
          "input": "base",
          "writes": [
            "patch"
          ]
        }
      },
      "metrics": {
        "iia": {
          "kind": "match",
          "of": "logits",
          "expected": "cf_answer",
          "token_form": "space_prefixed"
        },
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        }
      }
    },
    "ablation": {
      "reads": {
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "ablated",
          "input": "base"
        }
      },
      "writes": {
        "ablate": {
          "site": "target",
          "pos": -1,
          "mechanism": "swap",
          "operand": {
            "ref": "mean"
          }
        }
      },
      "models": {
        "ablated": {
          "input": "base",
          "writes": [
            "ablate"
          ]
        }
      },
      "metrics": {
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        }
      }
    }
  },
  "steps": {
    "clean": {
      "kind": "observe",
      "interventions": {
        "reads": {
          "acts": {
            "site": "target",
            "pos": -1,
            "model": "original",
            "input": "base"
          },
          "logits": {
            "site": "head",
            "pos": -1,
            "model": "original",
            "input": "base"
          }
        },
        "metrics": {
          "logit_diff": {
            "kind": "logit_diff",
            "of": "logits",
            "a": "cf_answer",
            "b": "base_answer",
            "token_form": "space_prefixed"
          }
        }
      },
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "outputs": {
        "mean": {
          "read": "acts",
          "reduce": "mean"
        }
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "clean.json"
        }
      ]
    },
    "compare": {
      "kind": "observe",
      "interventions": [
        "patching",
        "ablation"
      ],
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "saves": [
        {
          "value": "iia",
          "file_path": "iia.json"
        },
        {
          "value": "patching/logit_diff",
          "file_path": "logit_diff.json"
        },
        {
          "value": "ablation/logit_diff",
          "file_path": "logit_diff.json"
        }
      ]
    }
  }
}

documents/v2/two_observes.json — the steps block

  "steps": {
    "clean": {
      "kind": "observe",
      "interventions": {
        "reads": {
          "acts": {
            "site": "target",
            "pos": -1,
            "model": "original",
            "input": "base"
          },
          "logits": {
            "site": "head",
            "pos": -1,
            "model": "original",
            "input": "base"
          }
        },
        "metrics": {
          "logit_diff": {
            "kind": "logit_diff",
            "of": "logits",
            "a": "cf_answer",
            "b": "base_answer",
            "token_form": "space_prefixed"
          }
        }
      },
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "outputs": {
        "mean": {
          "read": "acts",
          "reduce": "mean"
        }
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "clean.json"
        }
      ]
    },
    "compare": {
      "kind": "observe",
      "interventions": [
        "patching",
        "ablation"
      ],
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "saves": [
        {
          "value": "iia",
          "file_path": "iia.json"
        },
        {
          "value": "patching/logit_diff",
          "file_path": "logit_diff.json"
        },
        {
          "value": "ablation/logit_diff",
          "file_path": "logit_diff.json"
        }
      ]
    }
  }

compare lists two declared interventions, so it runs each over its rows. A list is lowered, not executed specially: the step compiles to one nested plan per entry, named by the intervention, each holding the step exactly as a document naming only that intervention would compile it — so each writes into its own directory, compare/patching/ and compare/ablation/. A save on a list step names a value by its bare name when one entry produces it (iia is patching's alone), and as <intervention>/<value>, the nested plan's path, when several do. A name no entry has, or several have, is refused with the qualified names it could have been:

documents/v2/two_observes.json — with the compare saves reduced to a bare logit_diff

$ uv run causalab-mini validate bad/ambiguous_save.json
ValidationError: 1 validation error for Spec
  Value error, step 'compare': save 'logit_diff' is produced by 2 of its interventions ['patching', 'ablation']; qualify it as one of ['patching/logit_diff', 'ablation/logit_diff'] [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/value_error

clean writes its intervention in place: an entry of interventions may be the object itself instead of a name. It is compiled exactly as a declared one, because at load it is declared — under the step's own name, clean, or as <step>[i] when it is entry i of a list. It may not take a name that is already declared.

"No intervention" is not a missing field or a null. It is an intervention with no writes, whose reads say "model": "original": one un-intervened forward, scored by the same metrics as the others. clean is its own step rather than a third entry of the list because it also publishes the mean that ablation swaps in, through the outputs and ref seam described below. A nested plan reads what was published before it, and what it publishes stays its own, so a list entry cannot hand a value to its sibling.

$ uv run causalab-mini explain documents/v2/two_observes.json
<root>: Plan
  clean: Observe  metrics=['logit_diff/logit_diff']  outputs=['mean=mean(acts)']  saves=['clean.json']
      forward 'original' on 'base'  (4 rows x 11 tokens)
        read  'acts' at block_output[0] pos={index:-1} via 'identity'
        read  'logits' at lm_head pos={index:-1} via 'identity'
  compare: Plan
    patching: Plan
      compare: Observe  metrics=['iia/match', 'logit_diff/logit_diff']  saves=['iia.json', 'logit_diff.json']
          forward 'original' on 'counterfactual'  (4 rows x 11 tokens)
            read  'v_cf' at block_output[0] pos={index:-1} via 'identity'
          forward 'patched' on 'base'  (4 rows x 11 tokens)
            write 'patch' at block_output[0] pos={index:-1} swap(v_cf) via 'identity'
            read  'logits' at lm_head pos={index:-1} via 'identity'
    ablation: Plan
      compare: Observe  metrics=['logit_diff/logit_diff']  saves=['logit_diff.json']
          forward 'ablated' on 'base'  (4 rows x 11 tokens)
            write 'ablate' at block_output[0] pos={index:-1} swap(mean) via 'identity'
            read  'logits' at lm_head pos={index:-1} via 'identity'

model #

field	type	default	what it is
`key`	string	required	the HuggingFace repository id
`revision`	string	required	a commit, a branch or a tag. Part of the identity of the experiment, not a convenience
`dtype`	`"fp32"` \| `"bf16"`	required	what the weights are loaded as
`attn_implementation`	`"eager"` \| `"sdpa"` \| `null`	`null`	absent, the library's default. `eager` is the only one under which `attention_scores` and `attention_probs` exist

documents/real/weekdays_layer_sweep.json — the model block

  "model": {
    "key": "meta-llama/Llama-3.2-1B",
    "revision": "4e20de362430cd3b72f300e6b0f18e50e7166e08",
    "dtype": "fp32"
  }

attn_implementation is part of the experiment rather than of the run: the implementations differ in the last bits, and a document that reads an attention pattern has to ask for eager. There is no device field. Where the weights sit is a property of the run, and is a flag — see what is not a document field.

roles, and the rows a step runs over #

A role is one side of a paired prompt. Its only field is field, the column of a row its text comes from. Which rows it reads is not here: that is a step's, because two steps may run the same experiment over different splits.

documents/v2/patching.json — the roles block

  "roles": {
    "base": {
      "field": "input"
    },
    "counterfactual": {
      "field": "counterfactual_inputs[0]"
    }
  }

A field is a path into the row: input is a column, counterfactual_inputs[0] one entry of a list-valued column, and a.b walks into a dict. The value has to be a string.

A step names its rows as a dataset ref: a path relative to the data root (--data-root, default documents/data), with an optional #split. A table is a JSON array of flat row objects and a split is a column of that table selected by value, not a second file — so weekdays/data#train is the rows of documents/data/weekdays/data.json whose split is "train".

$ uv run causalab-mini data weekdays/data
weekdays/data: 4 rows
  columns: answer, base_answer, cf_answer, counterfactual_inputs, counterfactual_inputs_variables, entity, input, label, split
  splits:  test, train
  sample:  {"answer": " Friday", "base_answer": " Friday", "cf_answer": " Saturday", "counterfactual_inputs": ["If today is Friday, tomorrow is"], "counterfactual_inputs_variables": [{"entity": "Friday"}], "entity": "Thursday", "input": "If today is Thursday, tomorrow is", "label": " Saturday", "split": "train"}

`<column>_variables` #

A position may anchor to a variable — "this row's entity" — and each role then resolves its own text for it. The rule is the protocol's: the <column>_variables sibling of the role's own field first, then a top-level column of that name. For a role whose field is counterfactual_inputs[0], the variable entity is looked for at counterfactual_inputs_variables[0].entity, then at the column entity. That is what lets one position spec mean the right thing in both prompts of a pair. A name that is neither is refused on the client, naming both places it looked.

The forms a position can take are their own page; what matters here is that the anchor is a fact about the row, and so is resolved before anything runs.

sites #

field	type	default	what it is
`component`	string	required	one of the 21 names `vocab` lists
`layers`	`[int]` \| `null`	`null`	a one-element band. A layerless component takes none
`heads`	`[int]` \| `null`	`null`	at a per-head tensor, the heads this site is
`units`	`[int]` \| `null`	`null`	anywhere with a known width, the single features this site is

documents/v2/patching.json — the sites block

  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  }

A site names heads or units, never both: they slice the same axis at two grains, and the site's width becomes theirs — len(heads) · head_dim, or len(units) — so a featurizer, a swap or a harvest there is of those and leaves the rest alone. layers is a one-element list; a band spanning several layers is one address and is not implemented. Which components exist, which are per-head, which are read-only and which need eager attention is the locations page.

featurizers #

field	type	default	what it is
`kind`	`subspace` \| `pca` \| `gate` \| `sae` \| `linear`	required	what the parameter set is
`k`	int > 0 \| `null`	`null`	the rank of a basis. A gate has none: its features are the units
`parametrization`	`"cayley"`	`"cayley"`	how a rotation is parametrized
`seed`	int \| `null`	`null`	the draw the initial basis comes from. Absent, the fit's seed, or 0
`file_path`	string \| `null`	`null`	load the parameter from a safetensors bundle instead of drawing it

The validator pairs the kind with its numbers. A subspace or a pca needs k; a gate takes neither k nor seed, because it masks every unit of its site and starts at θ = 0. pca, sae and linear exist only as a file — nothing draws or trains one — so each needs file_path. A loaded featurizer has no seed: its weights are its bytes.

documents/v2/pca_control.json — a basis loaded from what pca_harvest.json wrote

  "featurizers": {
    "rot": {
      "kind": "pca",
      "k": 4,
      "file_path": "documents/artifacts/pca.safetensors"
    }
  }

a loaded bundle is checked against the document

A rotation is a grid of numbers and nothing in the numbers says which model or which layer it came from. The safetensors header does. A bundle this project wrote carries a stamp — model key, revision, dtype, component, layer, kind, k, d, parametrization — and the load compares every key that is present against what this document declares, refusing by name on a mismatch. A bundle from elsewhere, a published SAE, has no stamp of ours, and then the shapes are all there is to check: its width must be this site's.

A featurizer does not say where it is. One name is one parameter set, so the site it acts at is derived from the reads and writes that name it — and a name used at two sites is refused, because one rotation cannot have two widths. The kinds themselves, and what each does to a read or a write, are on the ops page.

interventions #

An intervention is the experiment, declared once. Every step that runs a forward names the ones it runs, or writes them in place (see how a document fits together). That is why a fit and the run that scores it cannot drift apart — they are the same declaration over different rows.

field	type	default	what it is
`reads`	`{name: object}`	required	what comes out
`writes`	`{name: object}`	`{}`	what goes in
`models`	`{name: object}`	`{}`	an intervened model: a role, and the writes in force on it
`metrics`	`{name: object}`	`{}`	what to score
`decode`	int ≥ 0	`0`	tokens to generate after the prompt. 0 is one forward pass

reads #

field	type	default	what it is
`site`	string	required	a declared site
`pos`	position	required	where in the sequence. A bare integer is `{"index": i}`
`input`	string	required	a declared role: whose prompt this forward runs
`model`	string	`"original"`	`"original"`, or an intervened model of this intervention
`featurizer`	string	`"identity"`	read the activation in that parameter set's space
`view`	`"raw"` \| `"logits"`	`"raw"`	`logits` projects a residual-stream read through the final norm and head

view: "logits" is only for the residual stream — embeddings, block_input, block_output, ln_final — because that is what the final norm and head expect, and it cannot be combined with a featurizer. A layer sweep plus a token_prob metric on a logits view is the logit lens, as one document.

writes #

field	type	default	what it is
`site`	string	required	a declared site, not read-only
`pos`	position	required	where in the sequence
`mechanism`	enum	required	one of the six below
`operand`	string \| number \| `{"ref": …}` \| `null`	`null`	what the mechanism acts with
`featurizer`	string	`"identity"`	the space the mechanism acts in
`features`	`[int]` \| `null`	`null`	which coordinates of that space it touches; the rest pass through
`params`	`{string: number}`	`{}`	the mechanism's numbers

mechanism and operand are two fields

The protocol writes a mechanism as a key — {"swap": "v_cf"} — and a key that is itself the mechanism's name cannot be enumerated by a schema. The schema is what an agent reads, so here the mechanism is a value.

mechanism	operand	required params	optional params
`swap`	yes	—	—
`add_scaled`	yes	—	`scale`
`lerp`	yes	`t`	—
`gaussian`	no	`seed`	`scale`
`clamp`	no	at least one of `lo`, `hi`	—
`renormalize`	no	—	—

The validator checks the pair: a mechanism that takes no operand is refused one, a mechanism that needs an operand is refused without one, and params is checked by name both ways — missing and extra. gaussian draws its own noise, clamp only bounds, and renormalize measures against the pre-write value the seam supplies, so none of the three takes an operand.

An operand is one of four things:

spelling	what it is
`"v_cf"`	a read of this intervention
`{"ref": "mean"}`	a value an earlier sibling step published under `outputs`
`0.0`	a literal number — zero ablation
`null`	nothing, for a mechanism that takes none

documents/v2/steer_renormalize.json — an add_scaled with a cross-step operand, and a renormalize after it

      "writes": {
        "steer": {
          "site": "target",
          "pos": -1,
          "mechanism": "add_scaled",
          "operand": {
            "ref": "direction"
          },
          "params": {
            "scale": 4.0
          }
        },
        "restore": {
          "site": "target",
          "pos": -1,
          "mechanism": "renormalize"
        }
      }

The order of writes inside an intervened model is the order they apply. A renormalize restores the norm the other writes at its site changed, so it is refused unless it comes after at least one of them and last among them; first, or alone, it is the identity. What each mechanism does to the tensor is the ops page.

models #

An intervened model has two fields: input, a declared role, and writes, the names of the writes in force on it. It is how one document holds two models at once — the steered one and the same one without the renormalize, say — and a read names which of them it reads.

A read in one model may be the operand of a write in another, so the forwards are ordered by that dependency. The graph has to be acyclic; it is the schedule, and a cycle is refused by the compiler.

metrics #

Every metric has kind, of (the name of a read) and token_form, which is "space_prefixed" and is the only form. The rest of its fields name columns of the base rows, not literal strings: the answer is per row, so the document names a column and the table carries the text.

kind	its columns	unit	estimand
`match`	`expected`	fraction	`match/v1`
`logit_diff`	`a`, `b`	logit	`logit_diff/v1`
`cross_entropy`	`target`	nat	`cross_entropy/v1`
`token_logit`	`token`	logit	`token_logit/v1`
`token_prob`	`token`	probability	`token_prob/v1`

documents/v2/patching.json — the metrics of the one intervention

      "metrics": {
        "iia": {
          "kind": "match",
          "of": "logits",
          "expected": "cf_answer",
          "token_form": "space_prefixed"
        },
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        }
      }

of has to be a read of the same intervention, and that read has to be one position wide: a metric scores one position per row, so a window of several is refused with the width it found.

rows, and eligibility

A row whose value in a named column is null or empty is an excluded measurement — not a zero. Which rows a metric can be computed for is decided on the client, where the data is: the eligible rows are the ones where every column the kind names has a value, and that vector is compiled into the plan and written into the saved table. A column that no row has is a misspelling and is refused.

A metric none of the rows is eligible for is refused before anything runs: "a metric of nothing has no mean". An excluded row still appears in the output table, with value: null, eligible: false and a reason, so it can never shorten a denominator quietly.

{
  "example_id": "0",
  "metric": "logit_diff",
  "value": -0.02835773676633835,
  "eligible": true,
  "positions": null,
  "reason": "",
  "unit": "logit",
  "estimand_version": "logit_diff/v1",
  "produced_by": "a53cd70e47fa3a3f34c71dda53b00d72d528eba4495aae0dd2182719820acad9"
}

example_id is the row's example_id column if the table has one, and the zero-based row index otherwise.

decode: generating after the prompt #

decode on an intervention generates that many tokens after the prompt, greedily, on every forward of it. With it, a position may name the continuation frame — {"frame": "generated", "index": k} — and the intervention publishes <model>.generated, the generated ids, which a save may take to a .safetensors file.

documents/v2/generate_probe.json — the read in the continuation frame, and the bound

        "logits": {
          "site": "head",
          "pos": {
            "frame": "generated",
            "index": {
              "sweep": [
                0,
                1,
                2
              ]
            }
          },
          "model": "patched",
          "input": "base"
        }
…
      "decode": 3

the decode is a bound, not a stopping condition

The generation runs with min_new_tokens equal to max_new_tokens and no sampling: EOS is held off, so the loop runs to its bound and the batch stays rectangular. How far a row meant to generate is then a result rather than a ragged tensor — its continuation frame is cut at its first stop token, and a row that never stopped has no eos segment, which comes home as a reported reason.

A generated position without decode > 0 is refused, as is a step beyond the bound. A write in the continuation frame is narrower still: it has to name a step the decode has already reached — {"index": k} with k ≥ 0, or {"all": true} — and takes no scope, because every other cut of the continuation is only known once the decode has finished.

steps #

The steps run in the order written, and that order is checked. Three kinds, told apart by kind.

`kind`	what it does	its fields
`observe`	one pass over its rows	`rows`, `interventions`, `outputs`, `saves`
`fit`	trains featurizers, then scores held-out rows	`rows`, `interventions`, `params`, `objective`, `anneal`, `epochs`, `pairs`, `seed`, `optimizer`, `early_stop`, `eval`, `saves`
`weights`	publishes fitted parameters	`names`, `saves`

An observe or a fit gives rows as role → dataset ref, and every declared role needs one. Both require interventions: a name declared under the document's interventions, an intervention object written in place, or a list of either, which runs each over the step's rows (see how a document fits together). Leaving it out is refused with the step's name and the names it could have used:

documents/v2/das.json — with interventions removed from the score step

$ uv run causalab-mini validate bad/unnamed.json
ValidationError: 1 validation error for Spec
  Value error, step 'score' runs a forward and names no intervention; say which with `interventions` — declared: ['das'] — or write one there in place [type=value_error, input_value={'header': {'description'...: 'rot.safetensors'}]}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/value_error

saves #

A save is {"value": …, "file_path": …} and it sits on the step whose result it names. value has to be something that step produces — a metric of its intervention, one of its outputs, a <model>.generated, or a name a weights step publishes — and naming anything else is refused with the list of what it does produce.

A metric goes to a JSON table; an output or a generated-ids value is a tensor and needs a .safetensors path. The path is relative to --out — or, in a swept document, to the point's own directory, which is the only nesting the output tree has. Every step of one document writes into that same directory.

the field is saves, not save

nnsight mounts a .save() method onto object, so every pydantic model in a process that has imported nnsight inherits one, and a field called save would shadow it.

outputs, and `ref` across steps #

An observe step may publish values for the steps after it. An output has read (a read of its intervention), reduce and k:

`reduce`	what is published	`k`
`none` (default)	the read as it is, `rows × width`	not allowed
`mean`	one vector, averaged over rows	not allowed
`pca`	the top-`k` principal directions, a basis a later document loads as a `pca` featurizer	required

A bare string is shorthand for keeping the read unreduced: "outputs": {"active": "latents"} means {"read": "latents", "reduce": "none"}.

Mean ablation is the whole pattern in one document: one step harvests and publishes a mean, a later step swaps it in with {"ref": "mean"}. The mean never touches disk unless a save asks for it.

documents/v2/mean_ablation.json — the steps block; the write that uses the output is in interventions.ablated

  "steps": {
    "harvest": {
      "kind": "observe",
      "interventions": "harvest",
      "rows": {
        "base": "weekdays/train"
      },
      "outputs": {
        "mean": {
          "read": "acts",
          "reduce": "mean"
        }
      },
      "saves": [
        {
          "value": "mean",
          "file_path": "mean.safetensors"
        }
      ]
    },
    "clean": {
      "kind": "observe",
      "interventions": "clean",
      "rows": {
        "base": "weekdays/train"
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "clean.json"
        }
      ]
    },
    "ablated": {
      "kind": "observe",
      "interventions": "ablated",
      "rows": {
        "base": "weekdays/train"
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "ablated.json"
        }
      ]
    }
  }

$ uv run causalab-mini explain documents/v2/mean_ablation.json
<root>: Plan
  harvest: Observe  metrics=[]  outputs=['mean=mean(acts)']  saves=['mean.safetensors']
      forward 'original' on 'base'  (4 rows x 11 tokens)
        read  'acts' at block_output[0] pos={index:-1} via 'identity'
  clean: Observe  metrics=['logit_diff/logit_diff']  saves=['clean.json']
      forward 'original' on 'base'  (4 rows x 11 tokens)
        read  'logits' at lm_head pos={index:-1} via 'identity'
  ablated: Observe  metrics=['logit_diff/logit_diff']  saves=['ablated.json']
      forward 'ablated' on 'base'  (4 rows x 11 tokens)
        write 'ablate' at block_output[0] pos={index:-1} swap(mean) via 'identity'
        read  'logits' at lm_head pos={index:-1} via 'identity'

Three things are checked about a reference. It must name an output an earlier step published — the message lists what has been published so far. Two steps may not publish the same name. And an output may not share its name with a read, because an operand names one or the other.

The compiler then checks the shapes, because it knows them and the block would only find out from a shape error: an unreduced output has to have the same row count and the same window as the write that swaps it in; a mean over a ragged read is one vector and broadcasts anywhere; and a pca basis is refused as an operand outright — a basis is loaded as a featurizer, not written at a site.

fit #

field	type	default	what it is
`rows`	`{role: ref}`	required	the rows it fits on
`params`	`[string]`	required	the featurizers it trains
`objective`	`[[number, string]]`	required	Σ wᵢ·termᵢ, minimized
`epochs`	int > 0	required	passes over the rows
`pairs`	int > 0	required	rows per update
`seed`	int	`0`	the shuffle, and the initial draw of a featurizer with no seed of its own
`optimizer`	object	required	`name` (`"adamw"`), `lr`, `weight_decay` (default `0.0`)
`early_stop`	object	required	`metric`, `mode` (`"max"` \| `"min"`, default `"max"`), `patience` > 0
`eval`	object	required	the held-out pass: its own `rows`, and its own `saves`
`anneal`	`{gate: object}`	`{}`	a gate's temperature: `start`, `end`, geometric between
`interventions`	string \| object \| `[string \| object]`	required	which experiment it fits through: a declared name, one written in place, or a list, which fits through each in turn as its own nested plan
`saves`	`[object]`	`[]`	its own results: `train/loss`, `train/eval`

An objective term is a metric of the intervention, or <gate>.mask for a gate this fit trains — the mean of its soft mask, which is its L1 penalty. early_stop.metric has to be a metric. params has to be declared featurizers, and none of them may be a pca, sae or linear, which are fixed by definition. anneal may only name a gate this fit trains. The eval pass needs rows for every role, and its saves have to be things that pass produces.

If the fit's rows and the eval's rows are different refs, the compiler checks they are endpoint-disjoint and refuses with the number of shared rows. The same ref for both is the visible train-equals-test ablation, and is allowed.

a featurizer may not be used before it is trained

A step that runs a forward uses every featurizer its intervention names. If a later fit trains one of them, that earlier step scored an untrained rotation — a valid document with a silently wrong number. It is refused, with the fix in the message: move the step after the fit, or, for a deliberate untrained baseline, declare a second featurizer that no fit names.

weights #

A weights step has names — declared featurizers — and saves. It is what writes a fitted parameter to a safetensors bundle, stamped with the identity a later document's load is checked against. An sae or a linear may not be named: its tensors are the file it came from, and there is no fitted weight to publish.

sweep #

{"sweep": [a, b, c]} at any field means the document is three points: the same experiment three times, differing in that one field. A sweep is lowered on the client before anything is compiled, so a point is an ordinary document in every way — its own addresses, its own tokenization, its own digest — and nothing downstream knows a sweep happened.

spelling	what it is
`{"sweep": [0, 1, 2]}`	the literal values
`{"sweep": {"range": [0, 16]}}`	0, 1, … 15. `[start, stop]` or `[start, stop, step]`, as Python's
`{"sweep": [-4, -1], "as": "pos"}`	a named axis: every wrapper with the same `as` moves together

Several swept fields are their cross product, one point per combination. The named axis exists because a cross product is sometimes the wrong thing: a patch reads at a position and writes at the same one, and swept separately those are N² points, most of them reading one place and writing another. Naming the axis makes them one coordinate.

documents/real/weekdays_layer_sweep.json — a range over layers, and a named axis shared by the read and the write

  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        {
          "sweep": {
            "range": [
              0,
              16
            ]
          }
        }
      ]
    },
    "head": {
      "component": "lm_head"
    }
  }
…
        "v_cf": {
          "site": "target",
          "pos": {
            "sweep": [
              -4,
              -1
            ],
            "as": "pos"
          },
          "model": "original",
          "input": "counterfactual"
        }
…
        "patch": {
          "site": "target",
          "pos": {
            "sweep": [
              -4,
              -1
            ],
            "as": "pos"
          },
          "mechanism": "swap",
          "operand": "v_cf"
        }

Sixteen layers × two positions, and because pos is one named axis rather than two fields, that is 32 points and not the 64 a cross product would give. Each point is a directory named for the values it took — layers=12,pos=-1/ — in the order the fields appear in the document.

$ uv run causalab-mini explain documents/v2/logit_lens.json
<root>: Plan
  layers=0: Plan
    lens: Observe  metrics=['p_answer/token_prob', 'top1/match']  saves=['p_answer.json', 'top1.json']
        forward 'original' on 'base'  (4 rows x 11 tokens)
          read  'lens' at block_output[0] pos={index:-1} via 'identity' as logits
  layers=1: Plan
    lens: Observe  metrics=['p_answer/token_prob', 'top1/match']  saves=['p_answer.json', 'top1.json']
        forward 'original' on 'base'  (4 rows x 11 tokens)
          read  'lens' at block_output[1] pos={index:-1} via 'identity' as logits

Two places may not be swept: model and header. The engine loads the model once and every point runs against it, and those two are the identity of the run rather than a coordinate in it. Everything else the protocol allows here — at_once, cohorts, swept bundles — is refused by name.

header, and the digest #

header has one field, description, and it is free text. Every document in the corpus uses it to say what the experiment is for and what its numbers do and do not mean; it is the first thing worth reading about a document.

The digest is the identity of the experiment: the SHA-256 of the document with header removed, serialized with sorted keys and no whitespace. It is what every metric row is produced_by and what a saved featurizer is stamped with. Rewriting the description does not make a new experiment:

$ uv run python -c '
import json
from causalab_mini.plan.spec import Spec
raw = json.load(open("documents/v2/patching.json"))
a = Spec.model_validate(raw).digest
raw["header"]["description"] = "anything else at all"
b = Spec.model_validate(raw).digest
print(a); print(b); print("same:", a == b)'
ffd162b5011bc74abf2f400d71595f63849a393a1fcf78aa234d26bbc77e791e
ffd162b5011bc74abf2f400d71595f63849a393a1fcf78aa234d26bbc77e791e
same: True

The converse is the reason nothing is quietly ignored: a key the reader skipped would still be in the bytes the digest is taken over, so two runs that did the same thing would claim to be different experiments — or worse, a key that changes the experiment would load, do nothing, and be stamped as if it had.

The schema, and the vocabulary #

causalab-mini schema prints the JSON Schema of the plan-shaped document — pydantic's dump of the same classes that validate it, so the schema and the checker cannot disagree. It is 1,116 lines, and this site carries a copy: schema.json, generated by causalab-mini schema on 2026-09-23 and regenerated by running it again.

vocab is the other half — the names a document may use, on one screen:

$ uv run causalab-mini vocab
components:
  attention_query        interior  [heads]
  attention_key          interior  [heads]
  attention_scores       interior  [heads]  needs eager attention
  attention_probs        nnterp attention_probabilities  [heads]  needs eager attention
  attention_z            interior  [heads]
  input_ids              nnterp embeddings_input  (read-only)
  embeddings             nnterp embeddings_output
  block_input            nnterp layers_input
  attention_input_norm   nnterp attentions_norm_output
  attention_premix       nnterp attentions_premix  [heads]
  attention_output       nnterp attentions_output
  block_mid              nnterp layers_mid
  mlp_input_norm         nnterp mlps_norm_output
  mlp_input              nnterp mlps_input
  mlp_activation         nnterp mlps_activation
  mlp_neuron_output      nnterp mlps_neurons
  mlp_output             nnterp mlps_output
  block_output           nnterp layers_output
  ln_final               nnterp ln_final_output
  lm_head                nnterp lm_head_output
  logits                 nnterp logits
mechanisms:       add_scaled, clamp, gaussian, lerp, renormalize, swap
featurizer kinds: gate, linear, pca, sae, subspace (plus 'identity', never declared)
metric kinds:     match(expected), logit_diff(a, b), cross_entropy(target), token_logit(token), token_prob(token)
position forms:   exactly one cut: index=int, span=tuple[int, int], last=int, all=bool
                  scope: variable=str, segment=Literal['system', 'user', 'assistant', 'eos']
                  frame: Literal['prompt', 'generated']; a bare integer is {"index": i}; -1 is the last token

Both verbs take --json, like every verb, so an agent parses instead of scraping.

The everything document #

documents/v2/dbm.json names more distinct fields than any other document in the corpus: all three step kinds, a gate featurizer, a fit with an objective of two terms, an anneal, an early stop and its own held-out saves, and a sweep on one of the objective weights so that the document is the sparsity curve.

documents/v2/dbm.json — the whole document

{
  "header": {
    "description": "DBM on the tiny CPU Llama: learn WHICH units of layer 0's output carry the answer, as a binary mask, rather than which directions (compare das.json \u2014 the only differences are the featurizer's kind, one objective term and the anneal). The mask is soft, \u03c3(\u03b8/T), while the fit updates it and hard, \u03b8 > 0, whenever it is scored; `mask.mask` in the objective is the mean of the soft mask, the L1 pressure toward keeping few units. The L1 weight is swept, so the document is the sparsity curve: measured 2026-09-21, 4, 2, 1 and 0 of 16 units kept."
  },
  "model": {
    "key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
    "revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
    "dtype": "fp32"
  },
  "roles": {
    "base": {
      "field": "input"
    },
    "counterfactual": {
      "field": "counterfactual_inputs[0]"
    }
  },
  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  },
  "featurizers": {
    "mask": {
      "kind": "gate"
    }
  },
  "interventions": {
    "dbm": {
      "reads": {
        "v_cf": {
          "site": "target",
          "pos": -1,
          "model": "original",
          "input": "counterfactual",
          "featurizer": "mask"
        },
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "patched",
          "input": "base"
        }
      },
      "writes": {
        "patch": {
          "site": "target",
          "pos": -1,
          "featurizer": "mask",
          "mechanism": "swap",
          "operand": "v_cf"
        }
      },
      "models": {
        "patched": {
          "input": "base",
          "writes": [
            "patch"
          ]
        }
      },
      "metrics": {
        "iia": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        },
        "ce": {
          "kind": "cross_entropy",
          "of": "logits",
          "target": "cf_answer",
          "token_form": "space_prefixed"
        }
      }
    }
  },
  "steps": {
    "fit": {
      "kind": "fit",
      "interventions": "dbm",
      "rows": {
        "base": "weekdays/data#train",
        "counterfactual": "weekdays/data#train"
      },
      "params": [
        "mask"
      ],
      "objective": [
        [
          1.0,
          "ce"
        ],
        [
          {
            "sweep": [
              0.0001,
              0.002,
              0.01,
              0.05
            ]
          },
          "mask.mask"
        ]
      ],
      "epochs": 20,
      "pairs": 16,
      "seed": 0,
      "optimizer": {
        "name": "adamw",
        "lr": 0.05,
        "weight_decay": 0.0
      },
      "early_stop": {
        "metric": "ce",
        "mode": "min",
        "patience": 20
      },
      "eval": {
        "rows": {
          "base": "weekdays/data#test",
          "counterfactual": "weekdays/data#test"
        },
        "saves": [
          {
            "value": "iia",
            "file_path": "held_out_iia.json"
          }
        ]
      },
      "anneal": {
        "mask": {
          "start": 1.0,
          "end": 0.05
        }
      }
    },
    "score": {
      "kind": "observe",
      "interventions": "dbm",
      "rows": {
        "base": "weekdays/data#train",
        "counterfactual": "weekdays/data#train"
      },
      "saves": [
        {
          "value": "iia",
          "file_path": "iia.json"
        },
        {
          "value": "ce",
          "file_path": "ce.json"
        }
      ]
    },
    "weights": {
      "kind": "weights",
      "names": [
        "mask"
      ],
      "saves": [
        {
          "value": "mask",
          "file_path": "mask.safetensors"
        }
      ]
    }
  }
}

What the compiler refuses #

Every message below is the real one, from running a deliberately broken document. Each is a copy of a corpus document with one thing changed, named in the caption. validate and explain do not catch their own exceptions, so the message arrives at the end of a traceback; the traceback is trimmed here with a … and nothing else is.

These eleven need no model and no weights, so they are available while you are still writing:

documents/v2/patching.json — with pos misspelt position on the read v_cf

$ uv run causalab-mini validate bad/unknown_key.json
…
pydantic_core._pydantic_core.ValidationError: 1 validation error for Spec
interventions.patching.reads.v_cf.position
  Extra inputs are not permitted [type=extra_forbidden, input_value=-1, input_type=int]

documents/v2/patching.json — with the read's site misspelt

$ uv run causalab-mini validate bad/undeclared_site.json
…
pydantic_core._pydantic_core.ValidationError: 1 validation error for Spec
  Value error, interventions.patching: read 'v_cf': undeclared site 'targit' [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/patching.json — with the write moved to a site whose component is input_ids

$ uv run causalab-mini validate bad/read_only.json
…
  Value error, interventions.patching: write 'patch': 'input_ids' is read-only — the model's input, not an activation [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/patching.json — with the mechanism changed to lerp and no params

$ uv run causalab-mini validate bad/lerp_no_t.json
…
pydantic_core._pydantic_core.ValidationError: 1 validation error for Spec
interventions.patching.writes.patch
  Value error, mechanism 'lerp' needs params ['t'] [type=value_error, input_value={'site': 'target', 'pos':...erp', 'operand': 'v_cf'}, input_type=dict]

documents/v2/patching.json — with the operand naming a value no read of this intervention has

$ uv run causalab-mini validate bad/operand_missing.json
…
  Value error, interventions.patching: write 'patch': operand 'mean' is not a read of this intervention; a value from an earlier step is {"ref": …} [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/patching.json — with a metric's of pointing at a name that is not a read

$ uv run causalab-mini validate bad/metric_of.json
…
  Value error, interventions.patching: metric 'iia': `of` must be a read name, got 'v_cf_logits' [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/patching.json — with a save naming a metric that does not exist

$ uv run causalab-mini validate bad/save_missing.json
…
  Value error, step 'score': saves 'iia2', which it does not produce (it produces ['iia', 'logit_diff']) [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/patching.json — with view: "logits" on a read of lm_head

$ uv run causalab-mini validate bad/view_logits.json
…
  Value error, interventions.patching: read 'logits': view 'logits' projects the residual stream through the head, and 'lm_head' is not the residual stream (one of ['block_input', 'block_output', 'embeddings', 'ln_final']) [type=value_error, input_value={'header': {'description'...: 'logit_diff.json'}]}}}, input_type=dict]

documents/v2/das.json — with the score step moved before the fit

$ uv run causalab-mini validate bad/order.json
…
  Value error, step 'score' uses featurizer 'rot' before step 'fit' trains it, so it would run on the untrained parameter. Move it after the fit — or, for a deliberate untrained baseline, declare a second featurizer that no fit names [type=value_error, input_value={'header': {'description'...: 'rot.safetensors'}]}}}, input_type=dict]

documents/v2/das.json — with rot also named by the read at the head site

$ uv run causalab-mini validate bad/two_sites.json
…
  Value error, featurizer 'rot' is used at ['head', 'target']; one name is one site [type=value_error, input_value={'header': {'description'...: 'rot.safetensors'}]}}}, input_type=dict]

documents/v2/patching.json — with model.dtype wrapped in a sweep

$ uv run causalab-mini validate bad/sweep_model.json
…
causalab_mini.plan.sweep.SweepError: model.dtype is swept; a sweep may not change the model or the header — those are the identity of the run, not a coordinate in it

The next two need the model's config, which explain reads without loading any weights. validate passes both, because neither the layer count nor the width of a site is in the document:

documents/v2/patching.json — at layer 99 of the two-layer fixture

$ uv run causalab-mini explain bad/layer99.json
…
causalab_mini.plan.plan.PlanError: site 'target': layer 99 is outside the model's 2 layers

documents/v2/das.json — with k: 999 at a 16-wide site

$ uv run causalab-mini explain bad/k999.json
…
causalab_mini.plan.plan.PlanError: featurizer 'rot': k=999 is not a subspace of the 16-wide site 'target'

where a refusal lives

A check about one value is a validator on the node that holds it. A check about two pieces of the document — a read's site, a save's value, the order of the steps — is a cross-check on the whole Spec. A check that needs the loaded model — a layer number, a site's width, a row count, an attention mask's layout — is the compiler's, and raises PlanError. That is why validate is fast and offline, and why explain is the next thing to run.

The protocol format #

The other reader accepts the protocol's own JSON, and only the subset of protocol_version 3 that this project runs. Every other real protocol feature is refused by name at load, so a document that cannot be run fails loudly instead of running as something else.

level	what is accepted
top level	`header`, `model`, `data`, `method`
`method`	`sites`, `reads`, `writes`, `intervened_models`, `featurizers`, `metrics`, `train`, `save`

Inside that, it is narrower than the plan-shaped format on almost every axis: one mechanism (swap), one featurizer kind (subspace), one optimizer, one early-stop mode, two roles (base and counterfactual), one position form (an integer index), a site at one layer with no heads and no units, and one flat save list rather than a save per step. There are no steps: the order of execution is implied, so a document with a fit is fit, then observe, then weights.

It has no schema, and its refusal of an unknown key is written out by hand at each level rather than falling out of the type. Those are the two things pydantic does for the plan-shaped format, and the reason that format exists. validate says which reader took a document:

$ uv run causalab-mini validate documents/minimal_cpu.json
ok: documents/minimal_cpu.json is a valid protocol document

The nine documents at the top of documents/ are in this format; documents/v2/ and documents/real/ are plan-shaped. Write new documents in the plan-shaped one.

What is not a document field #

A document is the experiment. Where it runs, how much of it runs at once and where the results land are the run's, and are flags:

flag	default	what it decides
`--engine`	`nnterp`	`nnterp`, `hooks`, or `ndif` — the same document with no local weights
`--device-map`	`auto`	where the weights sit. Ignored by `ndif`, whose weights are the server's
`--batch-size`	every row at once	rows per model call; bounds memory, moves only the last bit
`--data-root`	`documents/data`	what a dataset ref is relative to
`--out`	`out`	the directory the save tree is written under

None of them enters the digest, which is the point: the same document scored locally and on NDIF is the same experiment, and the numbers are comparable because nothing about the run is in its identity. What is recorded is the run: run.json, written beside the results, carries the engine, whether it was remote, the batch size, the library versions, a digest of this package and the dtype the model was actually served in.

{
 "engine": "NNterpEngine",
 "remote": false,
 "batch_size": null,
 "versions": {
  "torch": "2.14.0",
  "transformers": "5.17.0",
  "nnsight": "0.8.1.dev125+ga8ee93782",
  "nnterp": "1.3.1.dev64+g0e4b401af",
  "pydantic": "2.13.5"
 },
 "causalab_mini": "76c8c9d0c0d1f3e0b8e2eaeb09564107d1c0a70cb9e8b7d6cecab09fdc92d8a1",
 "served_dtype": "torch.float32"
}