Use case

Entity patching

The pair-native interchange: patch this row's entity, whatever it is.

An interchange intervention has to say where. A layer sweep says -4, and that works because every prompt in its corpus is the same shape. This document says something else: the last token of this row's entity, whatever text that is and wherever it falls. The base row's entity is Thursday; the counterfactual's is Saturday; one spec addresses both.

The spec is {"index": -1, "scope": {"variable": "entity"}}. scope picks the run of tokens the position is about — here, the tokens of this row's entity text, found in its prompt. index: -1 then cuts the last one of that run. Two independent questions, two fields.

At a glance #

$ uv run causalab-mini explain documents/v2/entity_patch.json
<root>: Plan
  score: Observe  metrics=['iia/match', 'logit_diff/logit_diff']  saves=['iia.json', 'logit_diff.json']
      forward 'original' on 'counterfactual'  (4 rows x 11 tokens)
        read  'v_cf' at block_output[0] pos={index:-1 scope:{variable:entity}} via 'identity'
      forward 'patched' on 'base'  (4 rows x 11 tokens)
        write 'patch' at block_output[0] pos={index:-1 scope:{variable:entity}} swap(v_cf) via 'identity'
        read  'logits' at lm_head pos={index:-1} via 'identity'

The compiled plan still carries the spec, not an integer. Nothing here could be resolved without the tokenizer and the row, so nothing is: the plan carries the anchor text per row and the run turns each one into indices.

The data #

$ uv run causalab-mini data weekdays/train
weekdays/train: 4 rows
  columns: answer, base_answer, cf_answer, counterfactual_inputs, counterfactual_inputs_variables, entity, input, label, split
  splits:  all
  sample:  {"answer": " Friday", "base_answer": " Friday", "cf_answer": " Sunday", "counterfactual_inputs": ["If today is Saturday, tomorrow is"], "counterfactual_inputs_variables": [{"entity": "Saturday"}], "entity": "Thursday", "input": "If today is Thursday, tomorrow is", "label": " Sunday", "split": "all"}

Two columns matter for the anchor. entity is the base row's entity text; counterfactual_inputs_variables holds the counterfactual's. That pairing is what lets one spec serve two roles, and it is a naming rule rather than a second field:

the <column>_variables sibling

A role's text comes from a field — input, or counterfactual_inputs[0]. Asked for the variable entity, the row is searched at that field's _variables sibling first: counterfactual_inputs[0] and entity give counterfactual_inputs_variables[0].entity. A top-level column of the name is the fallback, which is what base hits. A name that is neither is refused on the client, naming both places it looked.

On this tokenizer the entities are not the same length, which is the point:

$ uv run causalab-mini tokens hf-internal-testing/tiny-random-LlamaForCausalLM " Thursday" " Friday" " Saturday" " Sunday"
' Thursday'              3 token(s)  <- not usable as a metric column
' Friday'                1 token(s)
' Saturday'              1 token(s)
' Sunday'                1 token(s)

The warning is about metric columns — a metric scores one token — and it does not apply to an anchor. An anchor of three tokens is fine; the index: -1 inside it is what makes the window one token wide on every row.

The document #

documents/v2/entity_patch.json — the document this page runs, verbatim

{
  "header": {
    "description": "The pair-native interchange: one position spec, two roles, and each row's entity is different text. `{'index': -1, 'scope': {'variable': 'entity'}}` is the last token of THIS row's entity \u2014 ' Thursday' is three tokens on this tokenizer and ' Friday' one, so the base and the counterfactual are addressed by what they say rather than by a number counted off the end. The window is one token on every row, because 'the last token of' is a question about tokens, which is what makes the swap land where documents/v2/entity_mean_ablation.json's whole-entity read cannot. A row whose entity text is not in its prompt has nothing to write, and the run refuses it by name."
  },
  "model": {
    "key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
    "revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
    "dtype": "fp32"
  },
  "roles": {
    "base": {
      "field": "input"
    },
    "counterfactual": {
      "field": "counterfactual_inputs[0]"
    }
  },
  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  },
  "interventions": {
    "patching": {
      "reads": {
        "v_cf": {
          "site": "target",
          "pos": {
            "index": -1,
            "scope": {
              "variable": "entity"
            }
          },
          "model": "original",
          "input": "counterfactual"
        },
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "patched",
          "input": "base"
        }
      },
      "writes": {
        "patch": {
          "site": "target",
          "pos": {
            "index": -1,
            "scope": {
              "variable": "entity"
            }
          },
          "mechanism": "swap",
          "operand": "v_cf"
        }
      },
      "models": {
        "patched": {
          "input": "base",
          "writes": [
            "patch"
          ]
        }
      },
      "metrics": {
        "iia": {
          "kind": "match",
          "of": "logits",
          "expected": "cf_answer",
          "token_form": "space_prefixed"
        },
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer",
          "token_form": "space_prefixed"
        }
      }
    }
  },
  "steps": {
    "score": {
      "kind": "observe",
      "interventions": "patching",
      "rows": {
        "base": "weekdays/train",
        "counterfactual": "weekdays/train"
      },
      "saves": [
        {
          "value": "iia",
          "file_path": "iia.json"
        },
        {
          "value": "logit_diff",
          "file_path": "logit_diff.json"
        }
      ]
    }
  }
}

Reading it in the order the compiler does:

model is the tiny random-weight Llama, pinned by revision. It runs on a CPU in seconds, and its numbers mean nothing — this page is about where the swap lands, not about what the swap does.
roles gives base the input column and counterfactual the first entry of counterfactual_inputs.
sites.target is block_output at layer 0; sites.head is lm_head.
interventions.patching reads v_cf from the unmodified model on the counterfactual prompt, at the anchored position, and writes it into the base run at the same spec — which resolves to a different token, because it is a different row's text.
metrics scores the logits read at the last token: iia against cf_answer, logit_diff between the two answers.
steps.score is one observation pass over the four rows of weekdays/train.

Running it #

$ time uv run causalab-mini run documents/v2/entity_patch.json --out out/entity
out/entity/document.json
out/entity/run.json
out/entity/iia.json
out/entity/logit_diff.json

real	0m8.971s

Nine seconds, most of it the model load. One plan and no sweep, so the saves land straight in out/entity.

Where it landed #

A metric table row carries the positions of the read it scores — here logits, at a plain -1:

$ head -12 out/entity/iia.json
[
 {
  "example_id": "0",
  "metric": "iia",
  "value": 0.0,
  "eligible": true,
  "positions": [
   10
  ],
  "reason": "",
  "unit": "fraction",
  "estimand_version": "match/v1",
…

why there are positions at all

A pass whose every position is a fixed integer resolves the same indices on every row and every run, and the document already says so — that table gets "positions": null, as the layer sweep's does. This pass has an anchored position in it, so the whole pass reports where it read and wrote, and which rows it could place.

The provenance that matters here is the anchored read and write, which the run keeps in the plan beside the metrics: the indices, and what those indices decode back to.

$ python resolve.py
v_cf ((6,), (6,), (6,), (6,)) ("' Saturday'", "' Sunday'", "'day'", "' Friday'")
patch ((6,), (6,), (6,), (6,)) ("'day'", "' Friday'", "' Saturday'", "' Sunday'")

resolve.py — what produced the two lines above

import json
from causalab_mini import plan
from causalab_mini.engine.engines.nnterp import NNterpEngine
from causalab_mini.plan.spec import Spec

raw = json.loads(open("documents/v2/entity_patch.json").read())
engine = NNterpEngine.load(Spec.model_validate(raw).model, device_map="cpu")
done = engine.execute(plan.build_request(raw, "documents/data", engine))
where = done.step("score", plan.Observe).results["positions"]
for name in ("v_cf", "patch"):
    print(name, where[name]["rows"], where[name]["tokens"])

row	base entity	write index	token written	counterfactual entity	read index	token read
0	Thursday	6	`'day'`	Saturday	6	`' Saturday'`
1	Friday	6	`' Friday'`	Sunday	6	`' Sunday'`
2	Saturday	6	`' Saturday'`	Thursday	6	`'day'`
3	Sunday	6	`' Sunday'`	Friday	6	`' Friday'`

Every index is 6, and no two rows addressed the same token. Row 0's entity is Thursday, three tokens, and the last of them is the piece 'day'; row 1's is Friday, one token, addressed whole. Read the two halves together and you have the swap: the piece 'day' of the base's Thursday receives what the counterfactual's Saturday held.

why the index is equal and the token is not

Rows are padded on the left, so the last token of the batch is index 10 on every row whatever the prompt's length. Counting back from there, the suffix , tomorrow is is the same tokens everywhere, which puts the entity's final token at index 6 on all four rows. What sits at index 6 differs, because the entity does.

The window is the tokens that overlap the entity's character span, so a value starting in the middle of a piece brings that whole piece: the thing being addressed is a token.

A row whose entity is not in its prompt #

An anchor is a search in the decoded text. If the text is not there, the window is empty and the run says which of three reasons it was: alignment_missing, alignment_ambiguous — the value occurs more than once, and no occurrence index is minted — or out_of_range.

$ uv run causalab-mini run documents/v2/entity_patch.json --data-root /tmp/holed --out /tmp/out
…
causalab_mini.plan.plan.PlanError: write 'patch' has nothing to write on row(s) {1: 'alignment_missing'}. A read may skip a row; a write may not — writing nothing somewhere is not an intervention, and the row would score as if one had happened. `alignment_ambiguous` is a value that is in the prompt more than once, and scoping the anchor is what makes it one

/tmp/holed is a copy of documents/data with row 1's entity column set to "Neptune". Nothing else changed, and the run stops before a number is written.

a read may skip a row; a write may not

The asymmetry is deliberate. A read with an empty window is an excluded measurement: the row stays in the table with eligible: false, no value and the reason, so it can never be read as a zero or silently shorten a denominator. A write with an empty window would give a row that scores as if it had been intervened on when it had not, so it is refused by name — at the write, where the positions are.

Taking that row out of the data is all it takes: the others score what they scored, because a row is a row of a batch and nothing about the swap depended on it.

The same anchor, every token of it #

index: -1 is not the only cut. {"all": true} over the same scope reads every token of the row's entity — one for Friday, three for Thursday — which makes the read ragged: a different width per row.

documents/v2/entity_mean_ablation.json — the ragged read, verbatim

{
  "header": {
    "description": "Mean ablation where the mean is taken over a RAGGED read: the tokens of each row's entity (`{'all': true, 'scope': {'variable': 'entity'}}` \u2014 the row's own entity text, located in its prompt \u2014 one token for Friday, three for Thursday), averaged over every entity token in the corpus, then swapped in at the answer position. A row whose entity text is not in its prompt would contribute nothing and stay a row; here every row has one."
  },
  "model": {
    "key": "hf-internal-testing/tiny-random-LlamaForCausalLM",
    "revision": "9fb191250dd56d0ba7ec9785a025ed29c03d5998",
    "dtype": "fp32"
  },
  "roles": {
    "base": {
      "field": "input"
    }
  },
  "sites": {
    "target": {
      "component": "block_output",
      "layers": [
        0
      ]
    },
    "head": {
      "component": "lm_head"
    }
  },
  "interventions": {
    "harvest": {
      "reads": {
        "acts": {
          "site": "target",
          "pos": {
            "all": true,
            "scope": {
              "variable": "entity"
            }
          },
          "input": "base"
        }
      }
    },
    "clean": {
      "reads": {
        "logits": {
          "site": "head",
          "pos": -1,
          "input": "base"
        }
      },
      "metrics": {
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer"
        }
      }
    },
    "ablated": {
      "reads": {
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "ablated",
          "input": "base"
        }
      },
      "writes": {
        "ablate": {
          "site": "target",
          "pos": -1,
          "mechanism": "swap",
          "operand": {
            "ref": "mean"
          }
        }
      },
      "models": {
        "ablated": {
          "input": "base",
          "writes": [
            "ablate"
          ]
        }
      },
      "metrics": {
        "logit_diff": {
          "kind": "logit_diff",
          "of": "logits",
          "a": "cf_answer",
          "b": "base_answer"
        }
      }
    }
  },
  "steps": {
    "harvest": {
      "kind": "observe",
      "interventions": "harvest",
      "rows": {
        "base": "weekdays/train"
      },
      "outputs": {
        "mean": {
          "read": "acts",
          "reduce": "mean"
        }
      },
      "saves": [
        {
          "value": "mean",
          "file_path": "entity_mean.safetensors"
        }
      ]
    },
    "clean": {
      "kind": "observe",
      "interventions": "clean",
      "rows": {
        "base": "weekdays/train"
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "clean.json"
        }
      ]
    },
    "ablated": {
      "kind": "observe",
      "interventions": "ablated",
      "rows": {
        "base": "weekdays/train"
      },
      "saves": [
        {
          "value": "logit_diff",
          "file_path": "ablated.json"
        }
      ]
    }
  }
}

It harvests acts over every entity token in the corpus, averages them and swaps that mean in at the answer position — so the mean is over entity tokens rather than over a fixed slot. A ragged read comes home flat, with the run's own positions saying where each row's tokens were. A ragged write is a different matter: its operand has to be the same width per row, and a mismatch is refused naming the rows that differ.

What this page does not show #

the numbers here are noise

hf-internal-testing/tiny-random-LlamaForCausalLM has random weights and a 16-wide residual stream. It does not do the weekday task at all: the IIA above is 0.0 on every row, and it would be 0.0 whatever you patched. What the run shows is that the anchored spec resolved, per row, to the token it claimed — a fact about the resolver, which holds on any model. For an interchange result that means something, see the layer sweep, whose corpus is uniform enough to use a plain integer.