The CLI

Two kinds of verb. Seven of them — schema, vocab, model, tokens, data, validate and explain — need no weights. They say what exists, what a document means and whether it would compile, in seconds on a laptop, against a model that may only ever run on someone else's GPU. run is the eighth, and the only one that loads the weights.

Every verb takes --json, so an agent parses instead of scraping. Every verb writes to stdout and nothing else; only run touches the disk.

At a glance #

The loop is check, read, run. All three commands below take the same document, the smallest interchange in the repository: four rows, one swap at the last token of layer 0 of a tiny random-weight Llama. Documents is the page for what is in it.

$ uv run causalab-mini validate documents/v2/patching.json
ok: documents/v2/patching.json is a valid plan-shaped document, 1 step

$ uv run causalab-mini explain documents/v2/patching.json
<root>: Plan
  score: Observe  metrics=['iia/match', 'logit_diff/logit_diff']  saves=['iia.json', 'logit_diff.json']
      forward 'original' on 'counterfactual'  (4 rows x 11 tokens)
        read  'v_cf' at block_output[0] pos={index:-1} via 'identity'
      forward 'patched' on 'base'  (4 rows x 11 tokens)
        write 'patch' at block_output[0] pos={index:-1} swap(v_cf) via 'identity'
        read  'logits' at lm_head pos={index:-1} via 'identity'

$ uv run causalab-mini run documents/v2/patching.json --device-map cpu --out out/cli/patch
out/cli/patch/document.json
out/cli/patch/run.json
out/cli/patch/iia.json
out/cli/patch/logit_diff.json

The whole run took 7.9 s on twelve CPU cores, model load included. run prints the paths it wrote, in the order it wrote them, and that is its entire output.

The verbs #

The options are few. model and tokens take --revision (default main); model also takes --dtype, one of fp32 and bf16. data, validate, explain and run take --data-root, which defaults to documents/data — a relative path, so these commands expect to be run from the repository root.

Asking what exists #

vocab is the vocabulary a document is written in: every component you may name, every write mechanism, every featurizer and metric kind, and the forms a position may take. It loads nothing.

$ uv run causalab-mini vocab
components:
  attention_query        interior  [heads]
  attention_key          interior  [heads]
  attention_scores       interior  [heads]  needs eager attention
  attention_probs        nnterp attention_probabilities  [heads]  needs eager attention
  attention_z            interior  [heads]
  input_ids              nnterp embeddings_input  (read-only)
…
  block_output           nnterp layers_output
  ln_final               nnterp ln_final_output
  lm_head                nnterp lm_head_output
  logits                 nnterp logits
mechanisms:       add_scaled, clamp, gaussian, lerp, renormalize, swap
featurizer kinds: gate, linear, pca, sae, subspace (plus 'identity', never declared)
metric kinds:     match(expected), logit_diff(a, b), cross_entropy(target), token_logit(token), token_prob(token)
position forms:   exactly one cut: index=int, span=tuple[int, int], last=int, all=bool
                  scope: variable=str, segment=Literal['system', 'user', 'assistant', 'eos']
                  frame: Literal['prompt', 'generated']; a bare integer is {"index": i}; -1 is the last token

schema prints the JSON Schema pydantic derives from Spec, which is the plan-shaped format itself: 1,116 lines, 22 KB, every field with the docstring that explains it. It is too long to read as text and exactly the right size for a tool. Its definitions are the document's nouns:

$ uv run causalab-mini schema | jq -r '."$defs" | keys | join(", ")'
Anchor, Anneal, CrossEntropy, EarlyStop, Evaluation, Featurizer, Fit, Header, IntervenedModel, Intervention, LogitDiff, Match, Model, Observe, Optimizer, Output, Read, Reference, Role, Save, Site, TokenLogit, TokenProb, Weights, Where, Write

model is the same question asked of a checkpoint: how many layers it has, which side it pads on, and which of the components in vocab actually resolve on it — with each one's width, which is what a featurizer has to match. It needs the config, not the weights.

$ uv run causalab-mini model hf-internal-testing/tiny-random-LlamaForCausalLM
hf-internal-testing/tiny-random-LlamaForCausalLM via nnterp: 2 layers, padding_side=left
  attention_query    ok   width=16  op=attention_interface_1
  attention_key      ok   width=16  op=attention_interface_1
  attention_scores   --   component 'attention_scores' exists only under 'eager' attention, and this model runs 'sdpa': say "attn_implementation": "eager" in the document's model block
  attention_probs    --   component 'attention_probs' exists only under 'eager' attention, and this model runs 'sdpa': say "attn_implementation": "eager" in the document's model block
  attention_z        ok   width=16  op=attention_interface_1
  input_ids          ok   no width (no featurizer here)
…
  mlp_activation     ok   width=64
  mlp_neuron_output  ok   width=64
  mlp_output         ok   width=16
  block_output       ok   width=16
  ln_final           ok   width=16
  lm_head            ok   width=32000
  logits             ok   width=32000

tokens answers one question a metric depends on: is this string one token? A match or logit_diff column is read at a single position, so a two-token answer cannot be scored there. The tiny fixture's tokenizer is a good demonstration of why you ask rather than assume.

$ uv run causalab-mini tokens hf-internal-testing/tiny-random-LlamaForCausalLM " Monday" " Tuesday" "Sunday"
' Monday'                1 token(s)
' Tuesday'               3 token(s)  <- not usable as a metric column
'Sunday'                 1 token(s)

data reads a dataset ref the way a document's rows block does, and reports what is in it. empty is the column that matters most: a null or blank cell is an excluded measurement for any metric naming that column, and it is better to know before the run than after.

$ uv run causalab-mini data weekdays/train
weekdays/train: 4 rows
  columns: answer, base_answer, cf_answer, counterfactual_inputs, counterfactual_inputs_variables, entity, input, label, split
  splits:  all
  sample:  {"answer": " Friday", "base_answer": " Friday", "cf_answer": " Sunday", "counterfactual_inputs": ["If today is Saturday, tomorrow is"], "counterfactual_inputs_variables": [{"entity": "Saturday"}], "entity": "Thursday", "input": "If today is Thursday, tomorrow is", "label": " Sunday", "split": "all"}

The empty: line appears only when something is empty. The same table with one row's cf_answer removed, under a copied --data-root:

$ uv run causalab-mini data weekdays/train --data-root out/cli/holed-data
weekdays/train: 4 rows
  columns: answer, base_answer, cf_answer, counterfactual_inputs, counterfactual_inputs_variables, entity, input, label, split
  empty:   cf_answer (1 rows)
  splits:  all
…

Checking a document: one question, printed or not #

validate and explain do the same work. Both load the model as a meta-device shell — the config, the module tree, the tokenizer, no weights — resolve every address, tokenize every prompt and compile the plan. explain prints it; validate says whether it compiled, and in which of the two formats the file is written. Both take --data-root and --engine, and both use them.

$ uv run causalab-mini validate documents/v2/patching.json
ok: documents/v2/patching.json is a valid plan-shaped document, 1 step

$ uv run causalab-mini validate documents/pos_sweep_cpu.json
ok: documents/pos_sweep_cpu.json is a valid protocol document, 3 steps

So a misspelled component fails at validate: a component name is a question about a model, and this verb has one.

$ uv run causalab-mini validate out/cli/typo.json
AddressError: component 'block_ouput' has no address here, and this model's internals have no accessor of that name either. Mini's are ['attention_input_norm', 'attention_key', 'attention_output', 'attention_premix', 'attention_probs', 'attention_query', 'attention_scores', 'attention_z', 'block_input', 'block_mid', 'block_output', 'embeddings', 'input_ids', 'lm_head', 'ln_final', 'logits', 'mlp_activation', 'mlp_input', 'mlp_input_norm', 'mlp_neuron_output', 'mlp_output']; add one to a model with RenameConfig(addresses={...})

A refusal is a message on stderr and exit 1, not a traceback. The entry point catches this package's own error types by name — PlanError, DocumentError, AddressError, TokenError, LocateError, DataError, EngineError, pydantic's ValidationError and nnterp's RenamingError — and prints what they say. Nothing generic is caught, so a missing file is still a traceback. --traceback, before the verb like --json, restores the whole thing.

One exception, and it is a run through --engine ndif. nnsight flattens whatever the server raised into RemoteError, which is not one of the nine, so a remote refusal comes back as a traceback — 45 lines of it, ending in the same text and the same exit code. When the server refuses has the transcript and what the fix would be.

run #

There is no --dry-run, no --verbose and no --force. explain is the dry run; the output is the paths; and nothing is ever refused for already existing — see the caution under the output layout.

--engine: three runtimes, one plan #

Compiling is a weightless shell for all three, so explain prints the same plan whichever engine you name — for documents/v2/patching.json the three outputs are identical here. What differs is which addresses a runtime accepts, and where the forward happens.

The hooks engine is a plain HuggingFace model with a forward hook on each module a tap names. A hook fires at a module's boundary and sees nothing inside it, so the four interior components — the query, the key, the scores and the attention output before the out-projection — are refused by name rather than approximated. Everything else it reaches, including logits: a hook is handed the module's whole value and nnterp's row says where the tensor is inside it, which for the model's own output is the logits field. model --engine hooks is how you find out without running anything:

$ uv run causalab-mini model hf-internal-testing/tiny-random-LlamaForCausalLM --engine hooks
hf-internal-testing/tiny-random-LlamaForCausalLM via hooks: 2 layers, sdpa attention, padding_side=left
  attention_query    --   component 'attention_query' is an interior — one argument of one call inside a module's forward — and a forward hook only sees the module's boundary. It is addressed through nnsight's `.source`, which has no hooks equivalent; run this document on the nnterp engine.
…
  block_output       ok   width=16
  ln_final           ok   width=16
  lm_head            ok   width=32000
  logits             ok   width=32000

Everything a hook can reach, it reaches identically. Running documents/v2/patching.json on both engines here produced byte-identical iia.json and logit_diff.json; the run took 6.7 s on hooks against 7.9 s on nnterp. The two run.json files differ in the engine name, and in that only the nnterp one has a served_dtype: that field is read off the model inside the session, and only the nnterp engine has one.

There is no flag that runs hooks remotely, and the engine refuses one by name if it is asked: a hook is a Python callable registered on a module object in this process, and NDIF runs the model in another one, so there is nothing here to ship.

--engine ndif loads the same nnterp shell with dispatch=False — module tree and tokenizer, no parameters — and executes the compiled plan inside model.session(remote=True). --device-map is not passed at all. The model key must name a model the server hosts, and the run record says "remote": true. Remote & NDIF is the page for it.

--batch-size #

A window of rows is a whole small pass: the same forwards in the same order over a slice of the rows, with each published value sliced the same way so a write still meets the operand of its own row. What the windows read is concatenated back in row order, and everything after that — metrics, eligibility, outputs, saves — sees one pass. With no --batch-size there is one window, which is the pass exactly as compiled.

It is a property of the run and not of the experiment, which is why it is absent from document.json and present in run.json. It is not free of consequence: a GEMM over fewer rows rounds differently. The four rows of documents/v2/patching.json, run whole, in twos and in threes:

At 2 both tables came out byte-identical to the unbatched ones here; at 3 — which does not divide the four rows — the last digits of one row move. Do not count on either: FINDINGS.md §18 records the same comparison across patching, windows, the logit lens, generation, heads, neurons, a ragged harvest and a DAS fit, at batch sizes 1 and 3, and what it pins is agreement to 1e-6, not bit-equality.

What it does not do is bound the memory of a fit's update. The loss is the mean over the concatenated rows, so every window's graph is alive until the backward; a fit's memory knob is its pairs. It does bound the held-out pass, which runs under no_grad.

What lands on disk #

The plan's save manifest, and nothing else. Every file that leaves a run is a saves entry in the document, plus the document.json every plan's directory carries and the run.json the root carries. One document with one plan writes straight into --out:

out/cli/patch/
├── document.json      the experiment, verbatim, as it was given
├── run.json           what ran it
├── iia.json           a metric table
└── logit_diff.json

A nested plan writes below its parent in a directory named by its step name, so a plan's path in the tree is its path on disk. A sweep is exactly that: one root plan whose children are one plan per point, named for the field and the value it took.

$ uv run causalab-mini run documents/pos_sweep_cpu.json --device-map cpu --out out/cli/sweep
out/cli/sweep/document.json
out/cli/sweep/run.json
out/cli/sweep/pos=-1/document.json
out/cli/sweep/pos=-1/iia.json
out/cli/sweep/pos=-1/logit_diff.json
out/cli/sweep/pos=-2/document.json
out/cli/sweep/pos=-2/iia.json
out/cli/sweep/pos=-2/logit_diff.json
out/cli/sweep/pos=-3/document.json
out/cli/sweep/pos=-3/iia.json
out/cli/sweep/pos=-3/logit_diff.json

The directory name is the swept field's own name and the value it took, or the axis name when the sweep is a named one. The root's document.json is the document you wrote, sweep wrappers and all — its pos is still {"sweep": [-1, -2, -3]}. Each point's document.json is that point: the wrapper replaced by one value. So each point has its own digest, and each point's metric rows say which one produced them.

run.json is written once, on the root, because the engine records what ran on the plan it was handed.

A step that trains a featurizer writes it as a safetensors bundle alongside the tables, with its identity stamped into the header — the thing a later document's file_path load checks before trusting the rotation:

$ uv run causalab-mini run documents/v2/das.json --device-map cpu --out out/cli/das
out/cli/das/document.json
out/cli/das/run.json
out/cli/das/held_out_iia.json
out/cli/das/iia.json
out/cli/das/ce.json
out/cli/das/rot.safetensors

{
 "__metadata__": {
  "kind": "subspace",
  "k": "8",
  "d": "16",
  "parametrization": "cayley",
  "site": "target",
  "component": "block_output",
  "layer": "0",
  "model_key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
  "model_revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
  "model_dtype": "fp32",
  "featurizer_dtype": "fp32",
  "produced_by": "11703a0d56a047e5d3fa157ef455534c8803e76b2058cc5f469a9358e065ff34",
  "engine": "causalab-mini"
 },
 "weight": {"dtype": "F32", "shape": [16, 8], "data_offsets": [0, 512]}
}

That is the safetensors header of rot.safetensors, read back from the file. The keys are grouped here for reading and the weight entry is on one line; the names and the values are the file's.

run.json, annotated #

{
 "engine": "NNterpEngine",
 "remote": false,
 "batch_size": null,
 "versions": {
  "torch": "2.14.0",
  "transformers": "5.17.0",
  "nnsight": "0.8.1.dev125+ga8ee93782",
  "nnterp": "1.3.1.dev64+g0e4b401af",
  "pydantic": "2.13.5"
 },
 "causalab_mini": "76c8c9d0c0d1f3e0b8e2eaeb09564107d1c0a70cb9e8b7d6cecab09fdc92d8a1",
 "served_dtype": "torch.float32"
}

Deliberately absent: no hostname, no timestamp, no absolute path. A record has to be byte-identical across machines to be worth comparing.

A metric table row, annotated #

A metric table is a JSON array of row objects, one file per metric, with the labels repeated on every row so that jq and a person can both read it. Two rows of one table — the second one excluded — from documents/v2/entity_patch.json run against a data root whose row 1 has no cf_answer:

{
 "example_id": "0",
 "metric": "logit_diff",
 "value": -0.02950894832611084,
 "eligible": true,
 "positions": [
  10
 ],
 "reason": "",
 "tokens": "' is'",
 "unit": "logit",
 "estimand_version": "logit_diff/v1",
 "produced_by": "67d6465731d362667958723aa1faa600aca13bd2e4f78f73b895d38972af260c"
},
{
 "example_id": "1",
 "metric": "logit_diff",
 "value": null,
 "eligible": false,
 "positions": [
  10
 ],
 "reason": "",
 "tokens": "' is'",
 "unit": "logit",
 "estimand_version": "logit_diff/v1",
 "produced_by": "67d6465731d362667958723aa1faa600aca13bd2e4f78f73b895d38972af260c"
}

Row 1 above is the column half: its cf_answer is null, so logit_diff has nothing to score, and value is null with eligible: false. Its positions is still [10], because the position resolved fine — it was the data that was missing, and the two halves are reported separately for that reason.

--json, for agents #

Each verb returns one dictionary with a text key. --json prints the dictionary; without it, only text is printed. So the human output and the machine output are the same run, never two code paths that can disagree.

--json is on the top-level parser, so it goes before the verb:

$ uv run causalab-mini --json validate documents/v2/patching.json
{
 "text": "ok: documents/v2/patching.json is a valid plan-shaped document",
 "ok": true,
 "format": "plan-shaped"
}

$ uv run causalab-mini validate documents/v2/patching.json --json
usage: causalab-mini [-h] [--json]
                     {schema,vocab,model,tokens,data,validate,explain,run} ...
causalab-mini: error: unrecognized arguments: --json

The payloads are the answers a document-writing agent needs, in a shape it can query:

$ uv run causalab-mini --json tokens hf-internal-testing/tiny-random-LlamaForCausalLM " Monday" " Tuesday" \
    | jq -c '.texts | to_entries[] | {text: .key, one_token: .value.one_token}'
{"text":" Monday","one_token":true}
{"text":" Tuesday","one_token":false}

$ uv run causalab-mini --json model hf-internal-testing/tiny-random-LlamaForCausalLM \
    | jq -r '.components | to_entries[] | select(.value.resolves | not) | .key'
attention_scores
attention_probs

$ uv run causalab-mini --json vocab | jq -c '.metric_kinds, .units.match, .components.block_output'
{"match":["expected"],"logit_diff":["a","b"],"cross_entropy":["target"],"token_logit":["token"],"token_prob":["token"]}
{"unit":"fraction","estimand_version":"match/v1"}
{"accessor":"layers_output","path":null,"side":"output","interior":false,"layered":true,"seq_axis":1,"width":"hidden_size","read_only":false,"heads":false,"needs":null}

Two payloads are thin. explain adds only steps, the names of the root plan's steps — the plan itself is the text, and it is written to be read. On a swept document those names are the points, which is how you learn the directory names before the run makes them:

$ uv run causalab-mini --json explain documents/pos_sweep_cpu.json | jq -c '.steps'
["pos=-1","pos=-2","pos=-3"]

run adds written, the list of paths; the results are in those files, not in the reply.

$ uv run causalab-mini --json run documents/v2/patching.json --device-map cpu --out out/cli/patch
{
 "text": "out/cli/patch/document.json\nout/cli/patch/run.json\nout/cli/patch/iia.json\nout/cli/patch/logit_diff.json",
 "written": [
  "out/cli/patch/document.json",
  "out/cli/patch/run.json",
  "out/cli/patch/iia.json",
  "out/cli/patch/logit_diff.json"
 ]
}

Exit codes, and what an error looks like #

There is no error handling in the command line at all: a refusal is an exception, and it arrives with its stack. That is deliberate — the frames say which of the compiler, the address layer, the walk or the engine refused — but it means the sentence you want is the last line. Three real ones, each trimmed to its first line and its last, which is where the sentence is. This one exits 1:

$ uv run causalab-mini run documents/attention_query_cpu.json --engine hooks --device-map cpu --out out/cli/q-hooks
Traceback (most recent call last):
…
causalab_mini.address.AddressError: component 'attention_query' is an interior — one argument of one call inside a module's forward — and a forward hook only sees the module's boundary. It is addressed through nnsight's `.source`, which has no hooks equivalent; run this document on the nnterp engine.

A document that does not match the schema fails in pydantic, which names the field by its path:

$ uv run causalab-mini validate out/cli/badmetric.json
Traceback (most recent call last):
…
pydantic_core._pydantic_core.ValidationError: 1 validation error for Spec
interventions.patching.metrics.iia.match.expected
  Field required [type=missing, input_value={'kind': 'match', 'of': '...form': 'space_prefixed'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.13/v/missing

And a refusal that only the run can make, because only the run has the rows. A read may skip a row it cannot place; a write may not:

$ uv run causalab-mini run documents/v2/entity_patch.json --data-root out/cli/neptune-data --device-map cpu --out out/cli/neptune
Traceback (most recent call last):
…
causalab_mini.plan.plan.PlanError: write 'patch' has nothing to write on row(s) {1: 'alignment_missing'}. A read may skip a row; a write may not — writing nothing somewhere is not an intervention, and the row would score as if one had happened. `alignment_ambiguous` is a value that is in the prompt more than once, and scoping the anchor is what makes it one

The same hole on a read is not an error. It comes home as a row of the table with eligible: false, positions: [] and reason: "alignment_missing" — which is the case tests/test_eligibility.py pins. Whether a row the run cannot place stops the run or is reported depends on whether something was to be written there, not on the data.