Positions

A position is never an integer in a document. It is a spec with three independent fields — which sequence (frame), which run of it (scope), and how much of that run (exactly one of index, span, last and all) — and the integers come out of a row. One spec is written once and every row answers it for itself, which is what lets {"index": -1, "scope": {"variable": "entity"}} mean the last token of this row's entity, whatever text that is and however many tokens it takes.

Resolution happens inside the session, where the model is, against the model's tokenizer — not on the client. On a remote run that is the served checkpoint's own tokenizer, reached through nnsight's persistent objects, so no tokenizer bytes travel and a document runs against a model whose tokenizer the client never loads (FINDINGS §25.1). The resolver, causalab_mini/ops/locate.py, imports nothing but the standard library, torch and shapes.py for exactly that reason.

At a glance #

$ uv run causalab-mini vocab
…
position forms:   exactly one cut: index=int, span=tuple[int, int], last=int, all=bool
                  scope: variable=str, segment=str
                  frame: Literal['prompt', 'generated']; a bare integer is {"index": i}; -1 is the last token

Three positions out of one shipped document — the same spec on both sides of an interchange, and the bare integer that is the whole of the sugar:

      "reads": {
        "v_cf": {
          "site": "target",
          "pos": {
            "index": -1,
            "scope": {
              "variable": "entity"
            }
          },
          "model": "original",
          "input": "counterfactual"
        },
        "logits": {
          "site": "head",
          "pos": -1,
          "model": "patched",
          "input": "base"
        }
      },
      "writes": {
        "patch": {
          "site": "target",
          "pos": {
            "index": -1,
            "scope": {
              "variable": "entity"
            }
          },
          "mechanism": "swap",
          "operand": "v_cf"
        }
      },

The three questions #

They are independent: any cut of any scope in either frame. The resolver answers them in that order — find the run, cut it, check the cut is inside it — and a position with no scope is those same three steps with the run being the row's whole content. That is why {"index": -1} and {"index": -1, "scope": {"variable": "entity"}} cost the same code.

The cut #

The one sugar is that a bare integer is {"index": i}, so "pos": -1 is the last prompt token. There is no other spelling: a decode step is {"frame": "generated", "index": k}, in the same vocabulary as everything else.

Negatives count from the end of the run, not of the row. Inside a scope, {"index": -1} is the anchor's last token; with no scope it is the prompt's. A span may mix signs, and then its width is not knowable before the row is (shapes.py:142-143).

What you can ask for #

Chat turns #

A segment is a run of the row's prompt that the frame located, and in the prompt frame those are the turns of a conversation. What makes a role a chat is its data: a role's field may hold a string, or a list of {"role", "content"} messages, and a list is rendered through the checkpoint's own chat template when the document is compiled. There is no flag — the same shape an anchor already has, where what it looks for is whatever the row carries under that name.

  "chat": [
   {
    "role": "user",
    "content": "Name a weekday."
   },
   {
    "role": "assistant",
    "content": "Thursday."
   },
   {
    "role": "user",
    "content": "If today is Thursday, tomorrow is"
   }
  ],

The compiler records the character span of each turn's content, in the coordinates the resolver works in, and they travel in the plan. A turn is named by its own role, and by role[k] for the k-th of that role — the bracket a field path already uses for a list. The bare name is recorded only for a role that speaks once; asking for it when two did is alignment_ambiguous, the rule a variable occurring twice already gets.

forward.sample
'[INST] Name a weekday. [/INST] Thursday. </s><s>[INST] If today is Thursday, tomorrow is [/INST]'

forward.segments[0]
{"user[0]": [7, 22], "assistant[0]": [31, 40], "user[1]": [55, 88], "assistant": [31, 40]}

The template's own control tokens are between the turns and inside none of them, because each content is found as itself. So a read at the last token of the second user turn is the question's last token, where -1 of the whole prompt is the template's closing [/INST]:

{"index": -1, "scope": {"segment": "user[1]"}}   ((32,), (32,), (32,), (32,))   "' is'"
{"index": -1}                                    ((36,), (36,), (36,), (36,))   "']'"

{"segment": …, "variable": …} composes: the row's entity searched inside the turn that asked the question, which is what makes it unique when the assistant said the same word first. eos is the one segment name a frame invents rather than reads, and it belongs to the continuation; each frame refuses the other's.

Anchors: which text, on which row #

A variable anchor names a column, and the text it binds to is the row's own. The rule is the <column>_variables sibling of the role's field first, and a top-level column of that name otherwise: counterfactual_inputs[0] with the variable entity looks for counterfactual_inputs_variables[0].entity, then for a column entity. A name that is neither is a misspelling, refused on the client, naming both places it looked.

That rule is what lets one spec serve both roles of a pair. In weekdays/train the base role's field is input and picks up the top-level entity; the counterfactual role's field is counterfactual_inputs[0] and picks up the sibling:

$ uv run causalab-mini data weekdays/train
weekdays/train: 4 rows
  columns: answer, base_answer, cf_answer, counterfactual_inputs, counterfactual_inputs_variables, entity, input, label, split
  splits:  all
  sample:  {"answer": " Friday", "base_answer": " Friday", "cf_answer": " Sunday", "counterfactual_inputs": ["If today is Saturday, tomorrow is"], "counterfactual_inputs_variables": [{"entity": "Saturday"}], "entity": "Thursday", "input": "If today is Thursday, tomorrow is", "label": " Sunday", "split": "all"}

The compiler carries that text per row, in the plan, as the one input the resolver cannot find for itself: locate.py has no dataset. The spec and the per-row text travel together as Selection.where and Selection.anchors.

When the value is in the prompt twice, or not at all #

The search is in the decoded text, because decoding is what normalizes a sentencepiece prefix space. It tries the value, then " " + value, then the stripped value, and the first spelling that hits at all decides — so " Thursday" occurring once is one hit even though "Thursday" also occurs inside it. Anchoring to "day" on these prompts is ambiguous, because it is in today and in Thursday both — the last line of the block below.

The character map, and why the last token of Thursday is day #

The map from characters to tokens is built from the ids, by decoding growing prefixes: offsets[k] is the character at which token k starts. It needs no fast tokenizer and no offset_mapping, and it describes the ids the run will actually feed the model rather than a re-encoding that may not reproduce them.

from transformers import AutoTokenizer
from causalab_mini.ops import locate
from causalab_mini.shapes import Anchor, Where

tok = AutoTokenizer.from_pretrained(
    "hf-internal-testing/tiny-random-LlamaForCausalLM",
    revision="9fb191250dd56d0ba7ec9785a025ed29c03d5998")
ids, mask, frame = locate.frame_of_texts(
    tok, ["If today is Thursday, tomorrow is", "If today is Friday, tomorrow is"])

print(repr(frame.texts[0]))
print([tok.decode([i]) for i in ids[0][frame.starts[0]:frame.ends[0]]])
print(frame.offsets[0])

for where in (Where(all=True, scope=Anchor(variable="e")),
              Where(index=-1, scope=Anchor(variable="e"))):
    window, reason = locate.locate(frame, where, ("Thursday", "Friday"))
    print(where.spelling(), window,
          [locate.tokens_of(frame, one, row) for row, one in enumerate(window)], reason)

print(locate.locate(frame, Where(index=-1, scope=Anchor(variable="e")), ("day", "day")))

'If today is Thursday, tomorrow is'
['If', 'today', 'is', 'Th', 'urs', 'day', ',', 'tom', 'orrow', 'is']
(0, 2, 8, 11, 14, 17, 20, 21, 25, 30, 33)
{all scope:{variable:e}} ((4, 5, 6), (6,)) ["' Thursday'", "' Friday'"] ('', '')
{index:-1 scope:{variable:e}} ((6,), (6,)) ["'day'", "' Friday'"] ('', '')
(((), ()), ('alignment_ambiguous', 'alignment_ambiguous'))

A character span selects the tokens that overlap it, not the ones it contains, because the addressed thing is a token. The anchor "Thursday" starts at character 12 and the piece ' Th' starts at 11, so that piece is in the window and the window decodes back to ' Thursday'. Cut that run at -1 and you get its last token, which on this tokenizer is the piece day.

Position 0, and BOS #

A row's run starts after whatever the tokenizer puts before every prompt, and what that is, is asked rather than guessed: whatever the tokenizer makes of the empty string is what it adds to everything. Llama's sentencepiece prepends a BOS and GPT-2's BPE prepends nothing, so without this {"index": 0} would be <s> on one family and the first word on the other — the same document addressing two different things, silently, which is the one thing a standardized position may not do.

from transformers import AutoTokenizer
from causalab_mini.ops import locate
from causalab_mini.shapes import Where

FIXTURES = {
    "llama": ("hf-internal-testing/tiny-random-LlamaForCausalLM", "9fb191250dd56d0ba7ec9785a025ed29c03d5998"),
    "gpt2": ("hf-internal-testing/tiny-random-gpt2", "71034c5d8bde858ff824298bdedc65515b97d2b9"),
}
for family, (key, revision) in FIXTURES.items():
    tok = AutoTokenizer.from_pretrained(key, revision=revision)
    ids, mask, frame = locate.frame_of_texts(tok, ["If today is Thursday, tomorrow is"])
    window, _ = locate.locate(frame, Where(index=0))
    print(f"{family:6s} prefix={locate._special_prefix(tok)!s:5s} ids[:4]={ids[0][:4]} "
          f"starts={frame.starts} index 0 -> {window[0]} {locate.tokens_of(frame, window[0], 0)}")

llama  prefix=(1,)  ids[:4]=(1, 960, 9826, 338) starts=(1,) index 0 -> (1,) 'If'
gpt2   prefix=()    ids[:4]=(41, 70, 227, 68) starts=(0,) index 0 -> (0,) 'I'

The tiny GPT-2's vocabulary is small enough to split If in two, so what the test asserts is that the first token is the start of the prompt's own text and not a marker the tokenizer added (tests/test_locate.py:287-305).

The prefix itself is not addressable. It would be a run the frame locates, like eos in the continuation — {"scope": {"segment": "bos"}} — and no document has asked.

Left padding: the same index, a different token #

Resolution works from each row's half-open content span inside the padded sequence, so it does not care which side the padding is on. What left padding does mean is that every row's last token sits at the same index, and so does everything counted back from it. Rows that share a tail therefore share that tail's indices, and the index of the last token of whatever precedes it. The four prompts of entity_patch.json differ only in the weekday and all end , tomorrow is, so the last token of the entity is at index 6 on every row and no two rows address the same piece. The integers say nothing here and the decoded tokens say everything (see Provenance below for the run that produced them).

The index moves when the tokens after the anchor differ. On arithmetic prompts, where " Tuesday" is three tokens and " Monday" one, the same spec lands at 8 on two rows and at 6 on the other two (the same tok as above):

prompts = ["Q: What day is one days after Monday?\nA:",
           "Q: What day is three days after Monday?\nA:",
           "Q: What day is two days after Tuesday?\nA:",
           "Q: What day is five days after Tuesday?\nA:"]
_ids, _mask, frame = locate.frame_of_texts(tok, prompts)

windows, reasons = locate.locate(frame, Where(index=-1, scope=Anchor(variable="number")),
                                 ("one", "three", "two", "five"))
print(windows, reasons)
print([locate.tokens_of(frame, one, row) for row, one in enumerate(windows)])

((8,), (8,), (6,), (6,)) ('', '', '', '')
["' one'", "' three'", "' two'", "' five'"]

Both facts are worth knowing. The decoded tokens are what show an anchored patch doing something per row; an assertion that the integers differ needs a corpus where they do.

Eligibility: a read may skip a row, a write may not #

A row whose anchor text is not in its prompt gets an empty window. For a read that is an excluded measurement: the row contributes no positions, the run reports the reason, and it stays a row of the table. For a write it is refused — writing nothing somewhere is not an intervention, and the row would score as if one had happened. The message names each row and that row's own reason, because the three have three different fixes (tests/test_ragged.py:243-297 asserts both shapes):

write 'patch' has nothing to write on row(s) {1: 'alignment_missing'}. A read may
skip a row; a write may not — writing nothing somewhere is not an intervention, and
the row would score as if one had happened. `alignment_ambiguous` is a value that is
in the prompt more than once, and scoping the anchor is what makes it one

An empty window means two different things and they get opposite answers. An anchored cut that found nothing is data, reported per row. A cut of a fixed width that a row cannot fit — {"last": 12} on a nine-token row, {"index": 40} on any of these — names the same number of tokens on every row, so a row it does not fit is a document that is wrong about its own prompts. That is refused where it is resolved, naming the op, the rows and the reason.

A metric's rows are the intersection of two halves. The column half is the client's — whether the data had an answer to score. The position half is the run's — whether it could place the read. Neither is authoritative alone, and the run reports the intersection as results["eligible"]. A metric none of whose rows survive both halves is refused: a metric of nothing has no mean (steps.py:207-212).

Provenance #

A pass with a position the document does not already fix records where it acted. results["positions"] holds, per op, each row's rows window, the reason it is empty when it is, and the tokens it addressed — plain tuples of integers and strings, so they come home in the plan like a metric does.

A pass whose every position is a bare integer resolves the same numbers on every row and every run, and the document already says so: it builds no character map and reports nothing. The map is the expensive half — O(L) decode calls of O(L) work per row — and only a pass that reports where it acted has any use for it (locate.py:85-91).

$ uv run causalab-mini run documents/v2/entity_patch.json --out out/entity_patch
out/entity_patch/document.json
out/entity_patch/run.json
out/entity_patch/iia.json
out/entity_patch/logit_diff.json

A write's record has no table to live in, and for that the plan the run filled in is where to look. Save this as where.py:

import json
import sys

from causalab_mini import plan
from causalab_mini.engine import NNterpEngine
from causalab_mini.plan.spec import Spec

raw = json.loads(open(sys.argv[1]).read())
engine = NNterpEngine.load(Spec.model_validate(raw).model, device_map="cpu")
done = engine.execute(plan.build_request(raw, "documents/data", engine))
results = done.step("score", plan.Observe).results

for name, record in results["positions"].items():
    print(name, record["rows"], record["tokens"], record["reason"])
print("eligible", results["eligible"])

$ python where.py documents/v2/entity_patch.json
v_cf ((6,), (6,), (6,), (6,)) ("' Saturday'", "' Sunday'", "'day'", "' Friday'") ('', '', '', '')
patch ((6,), (6,), (6,), (6,)) ("'day'", "' Friday'", "' Saturday'", "' Sunday'") ('', '', '', '')
logits ((10,), (10,), (10,), (10,)) ("' is'", "' is'", "' is'", "' is'") ('', '', '', '')
eligible {'iia': (True, True, True, True), 'logit_diff': (True, True, True, True)}

Row 0's base entity is Thursday and its counterfactual's is Saturday: the write lands on the piece day where the read took ' Saturday' whole. Both are at index 6.

All three reach disk. Every row of a metric table carries positions — where the number was read — reason — why it was nowhere — and tokens, what that window says. The window is the one the metric's own read resolved to, so on this document it is the logits read at -1 and not the entity:

{
 "example_id": "0",
 "metric": "iia",
 "value": 0.0,
 "eligible": true,
 "positions": [
  10
 ],
 "reason": "",
 "tokens": "' is'",
 "unit": "fraction",
 "estimand_version": "match/v1",
 "produced_by": "67d6465731d362667958723aa1faa600aca13bd2e4f78f73b895d38972af260c"
}

An excluded measurement is still a row of the table, with "value": null and "eligible": false, so it can never be read as a zero or silently shorten a denominator.

The generated frame #

{"frame": "generated"} addresses what the decode produced. The decode runs to its bound — mini holds EOS off so the loop is a bound and the batch stays rectangular — but how far a row meant to generate is a result, so the frame is cut per row at that row's first stop token and the stop token is the last one in it. A row that never stopped has the whole bound and no eos segment.

Its character map is built from the ids the decode produced, never from re-encoding the finished text: a tokenizer is free to merge across a boundary the decode never saw.

{"index": k} with k >= 0 is decode step k, and the tap fires there. Every other cut of the continuation — the last generated token, the stop token, where the model said the answer — is of text that does not exist until the decode has run, so the compiler emits one ordinary read per step, each carrying the same spec and the name of the stack it belongs to, and the walk stacks them and cuts them against the continuation afterwards. Neither engine had to learn to buffer. The cost is stated where it is paid: rows × decode × width numbers, refused above 256 MiB (build.py:931-940, 951-965).

A write is the other case. It happens during the decode, so it may only name a step the run has already reached:

Every tap in this frame is reported by the continuation and not by the prompt, stacked or not: the prompt frame has nothing true to say about a position that is a decode step. So positions for a generated-frame op is the step, and tokens is the token the model produced there.

$ uv run causalab-mini run documents/v2/generated_answer.json --out out/generated_answer
out/generated_answer/document.json
out/generated_answer/run.json
out/generated_answer/p_answer.json

The same script as above, on that document:

$ python where.py documents/v2/generated_answer.json
at_said ((1,), (2,), (), ()) ("' substr'", "' energ'", '', '') ('', '', 'alignment_missing', 'alignment_missing')
at_stop ((), (), (), ()) ('', '', '', '') ('alignment_missing', 'alignment_missing', 'alignment_missing', 'alignment_missing')
eligible {'p_answer': (True, True, False, False)}

at_said is {"frame": "generated", "index": -1, "scope": {"variable": "said"}}: the last token of this row's own said text, in what this row generated. Two of the four rows said it — at step 1 and at step 2 — and two did not, and that is a reported result rather than an exception that ends the run. The table keeps all four rows:

[
 {
  "example_id": "substr",
  "metric": "p_answer",
  "value": 3.008824569405988e-05,
  "eligible": true,
  "positions": [
   1
  ],
  "reason": "",
  "unit": "probability",
  "estimand_version": "token_prob/v1",
  "produced_by": "8dcb96076c71ff1883ad97ecead2ece4ae24d447a121d013bb60e0155198f334"
 },
 …
 {
  "example_id": "Friday",
  "metric": "p_answer",
  "value": null,
  "eligible": false,
  "positions": [],
  "reason": "alignment_missing",
  "unit": "probability",
  "estimand_version": "token_prob/v1",
  "produced_by": "8dcb96076c71ff1883ad97ecead2ece4ae24d447a121d013bb60e0155198f334"
 },
 …
]

at_stop asks {"scope": {"segment": "eos"}} and every row reports alignment_missing. That is not the document being wrong: mini decodes with min_new_tokens == max_new_tokens so the loop is a bound, so no row ever stops, so nothing has an eos segment. "Did the model stop?" comes home as a reported reason. The positive case is reachable only over hand-made ids, which is where it is tested (tests/test_locate.py:149-164).

Width and ragged #

Two properties of a spec, both knowable without a row, answering different questions.

So {"index": -1, "scope": {"variable": "entity"}} has width 1 and is ragged: the cut names one token, and whether a row has that one token at all is the other question. {"all": true} has no width and is ragged for the first reason. width is what lets the compiler check a write against its operand before either has been resolved; ragged is what decides that the gather comes back flat, (total, width), one row per row that resolved — see Reads, writes, metrics for what a read and a write then do with it.

A metric reads width == 1 and nothing else. Ragged is fine: p_answer above reads a ragged width-1 window and scores the two rows it found. {"all": true} or {"last": 3} under a metric is refused at validation.

In the continuation frame ragged is about the steps a row keeps, not the positions a tap acts at: whatever the spec names, a decode step processes one position, and the cut happens afterwards over the stack of steps.

What is refused, and where #

That last check compares decoded strings rather than re-encoding, because encode(decode(ids)) == ids is not a property tokenizers have: it holds for ASCII on both fixtures here and fails on the first emoji. One row is a sample, so a skew that first shows on row 7 passes.