Remote & NDIF

--engine ndif changes one thing in the run: the one model.session(...) is opened with remote=True. The client still compiles the document — the widths, the module paths, an interior's operation, the token ids of every prompt and every metric column — against a meta-device shell with no weights in it. What comes out is a plan of strings, integers and tuples, and that is what travels. The server runs the block on the checkpoint it already has loaded, against its own installed copy of causalab_mini, and only what fills the plan in comes home.

At a glance #

$ NDIF_HOST=http://localhost:8001 causalab-mini run documents/v2/entity_patch.json --engine ndif --out out/entity-ndif
✓ [ae1d1705ad59466198dcd56bb1b92c9a] COMPLETED    (0.4s) Your job has been completed.
out/entity-ndif/document.json
out/entity-ndif/run.json
out/entity-ndif/iia.json
out/entity-ndif/logit_diff.json

The same four files a local run writes, with the same bytes in the two metric tables. In run.json exactly one field differs, and it is remote.

What runs where #

The tokenizer row is a rule, not an accident. A traced block ships every name it loads, so a block that closed over the client's tokenizer would carry that object to the server and resolve positions against it. The block reaches model.tokenizer instead — an attribute, and one of nnsight's persistent objects, which a server resolves to the served checkpoint's own. tests/test_structure.py reads the package's own source and fails if any trace body loads the bare name tokenizer, self, or a client-side module.

Pointing at a server #

mini has no host setting of its own. The host and the key are nnsight's, read from the environment at import:

Both can also live in $XDG_CONFIG_HOME/nnsight/config.yaml, which the environment overrides. What a server is serving, and at what dtype, is on its /status:

$ curl -s $NDIF_HOST/status | jq -r '.deployments[] | select(.deployment_level=="HOT") | [.repo_id, .dtype, .application_state] | @tsv'
hf-internal-testing/tiny-random-LlamaForCausalLM	float32	RUNNING
meta-llama/Llama-3.2-1B	float32	RUNNING

That listing had 553 entries; the other 551 are COLD, and a COLD record carries no dtype, because nothing has been loaded to have one. A document's model.key and model.revision have to match a record here, and its dtype is a request the server is free to ignore — see below.

The server needs the same packages #

A remote block is recompiled on the server and the modules it names are imported there. So the server must have causalab_mini and nnterp installed, at the versions the client has. /env says what it has:

$ curl -s $NDIF_HOST/env | jq -r '.python_version, (.packages | {torch, transformers, nnsight, nnterp, causalab_mini, pydantic} | to_entries[] | "\(.key) \(.value)")'
3.12.14 (main, Sep  1 2026, 00:10:07) [GCC 14.2.0]
torch 2.14.0+cu126
transformers 5.17.0
nnsight 0.8.1.dev125+g524c33fc
nnterp 0.5.0.dev0+internalsaccessors
causalab_mini 0.0.1
pydantic 2.13.4

A version string is coarse for a package installed from a checkout, so the sharper check is mini's own digest — a sha256 over every .py file in the package, which run.json already records. Read from inside a remote block against this stack, the server's install and the client's agree:

SERVER /usr/local/lib/python3.12/site-packages/causalab_mini/__init__.py
SERVER_DIGEST 76c8c9d0c0d1f3e0b8e2eaeb09564107d1c0a70cb9e8b7d6cecab09fdc92d8a1
CLIENT_DIGEST 76c8c9d0c0d1f3e0b8e2eaeb09564107d1c0a70cb9e8b7d6cecab09fdc92d8a1
MATCH True

The __file__ on the first line is the other half of it: the server's path, not the client's, which is what says the module was imported rather than rebuilt from the request.

What a mismatch looks like #

Nothing checks the two sides against each other. A skew shows up as a missing name, at the moment the block reaches for it — here, a client calling a function its server's copy does not have:

nnsight.intervention.backends.remote.RemoteError: Traceback (most recent call last):
…
  File "/usr/local/lib/python3.12/site-packages/ndif/services/ray/deployments/modeling/nns.py", line 183, in execute_traced_block
    tracer.execute(tracer.info.code)
  File "[skewed:25] …/probe_versions.py", line 27, in skewed
    return steps.a_function_added_after_the_server_was_built()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'causalab_mini.engine.steps' has no attribute 'a_function_added_after_the_server_was_built'

What the request weighs #

Measured with nnsight's own report (NNSIGHT_DEBUG=1 prints the payload and the result blob, both after compression), and with cloudpickle's by-value set read at serialize time:

The third row is the package travelling with the request: 13.6x the first, and the same ~64 KB of source on every request whatever the document asks. A plan is cheap to ship because it is only strings, integers and tuples — an anchored position costs a handful of them, and the code that resolves it is on the server already.

The dtype is the server's #

A document's model.dtype is a request. The deployment decides what it serves, and dtype is not part of the model key, so a client cannot ask for one and cannot see what it got from the outside. The block therefore reads model.lm_head.weight.dtype on the server and brings the string home, where it lands in run.json as served_dtype.

It matters more than a rounding note. Recorded on this stack, the same 32-point sweep run locally in fp32 and through NDIF, 2,688 values per comparison:

So "close, not equal" is entirely the served dtype. At fp32 every one of the 2,688 numbers is bit-identical to the local GPU run: neither the serialization round trip nor running in someone else's process perturbs the arithmetic. The three iia flips under bf16 are all at layers 12–13 at the last token, the two points where the sweep crosses over and the argmax between the two answers is nearly tied.

documents/real/README.md records the same comparison against a bf16 deployment on an A100: last-token IIA 0.86 where local reads 0.88, p(answer) differing by up to 0.018, L12 H28 at +0.051 against +0.054 — and the DAS rank curve identical.

What came home #

The layer sweep, which is the largest document in the repository — 32 experiments, 42 rows each, a read, a swap write and two metrics, all in one job:

$ NDIF_HOST=http://localhost:8001 causalab-mini run documents/real/weekdays_layer_sweep.json --engine ndif --out out/sweep-ndif
✓ [d7ac06dc7def47da8d16f3dd77f10276] COMPLETED    (4.6s) Your job has been completed.
out/sweep-ndif/document.json
out/sweep-ndif/run.json
out/sweep-ndif/layers=0,pos=-4/document.json
out/sweep-ndif/layers=0,pos=-4/iia.json
out/sweep-ndif/layers=0,pos=-4/logit_diff.json
…
out/sweep-ndif/layers=15,pos=-1/document.json
out/sweep-ndif/layers=15,pos=-1/iia.json
out/sweep-ndif/layers=15,pos=-1/logit_diff.json

Against the same document run locally on the GPU, fp32, everything but the run record is byte-for-byte the same file:

$ diff -r -x run.json out/sweep-local out/sweep-ndif && echo "ALL 64 FILES IDENTICAL"
ALL 64 FILES IDENTICAL

$ diff out/sweep-local/run.json out/sweep-ndif/run.json
3c3
<  "remote": false,
---
>  "remote": true,

One field. The server here is deployed at the document's own dtype, so served_dtype reads torch.float32 on both sides; against the default deployment it would be the second field to differ, and the only one that would move a number. The whole record:

{
 "engine": "NNterpEngine",
 "remote": true,
 "batch_size": null,
 "versions": {
  "torch": "2.14.0+cu126",
  "transformers": "5.17.0",
  "nnsight": "0.8.1.dev125+g524c33fc",
  "nnterp": "0.5.0.dev0+internalsaccessors",
  "pydantic": "2.13.5"
 },
 "causalab_mini": "d52d3c7757ac30d38e9990ba441d563756235e6b6d4312351110acec46185e73",
 "server": {
  "asked": "http://localhost:8001",
  "python": "3.12.14",
  "versions": {
   "torch": "2.14.0+cu126",
   "transformers": "5.17.0",
   "nnsight": "0.8.1.dev125+g524c33fc",
   "nnterp": "0.5.0.dev0+internalsaccessors",
   "pydantic": "2.13.4"
  }
 },
 "served_dtype": "torch.float32"
}

Those versions are read with importlib.metadata, in the client's process, which on a remote run is an assumption about a machine the record has never spoken to. A remote run asks: nnsight's /env lookup, cached per host, and what comes back is recorded under server beside them, filtered to the same five names so the two compare line for line. A server that will not answer is recorded as not having answered. asked is the host the client resolved, so a run against the public service reads "asked": "https://api.ndif.us". The same command with --engine nnterp writes the same file without the key: a local run has no server to ask.

Read the two pydantic lines. The client has 2.13.5 and the server 2.13.4, on a stack built to match — every other watched package agrees. That skew was invisible before this key existed, and finding it is what the key is for.

causalab_mini is a sha256 over every .py file in the package, and it is not in WATCHED because a version string is coarse for a package installed from a checkout. On this run the server computed it inside the block and it matched, from a different path:

server digest: d52d3c7757ac30d38e9990ba441d563756235e6b6d4312351110acec46185e73
client digest: d52d3c7757ac30d38e9990ba441d563756235e6b6d4312351110acec46185e73
server file  : /usr/local/lib/python3.12/site-packages/causalab_mini/engine/provenance.py
client file  : /home/localjadenfk/wd/causalab-mini/causalab_mini/engine/provenance.py

Two paths, one digest. That is the claim the record's causalab_mini field makes, checked end to end rather than assumed: the server ran its own installed copy and it is byte-identical to the client's.

Ragged results come home too, not just tensors. An anchored document's positions record — each row's window, its decoded tokens and its reason code — and the per-row eligible mask are nested dicts of integers and strings, and they arrive with nothing lost and nothing reordered. documents/v2/entity_patch.json, whose position is {"index": -1, "scope": {"variable": "entity"}}, writes the same positions column locally and remotely.

So does a chat document, whose spans are the client's alone: the compiler renders the conversation through the template and records where each turn landed, and those travel in the plan rather than being worked out on the server. documents/v2/chat_turn.json through NDIF is the local run leaf for leaf — all 46 of them — and {"index": -1, "scope": {"segment": "user[1]"}} comes back as ' is', the question's own last token, on both.

Timings #

The server-side figure is the model actor's own log line (prev_stage_ms between running and completed). The rest is the client: importing torch, building the meta shell, compiling 32 documents and tokenizing. That is why a remote run of a small document is not faster — the weights were never the slow part. The model was already HOT for both figures; a cold deployment adds its load time to the first request.

When the server refuses #

A document whose write has nothing to write on some row is refused at run time, because whether a row's entity text is in that row's prompt is not known until the prompt is tokenized. Locally the CLI catches it and prints what it says:

$ causalab-mini run entity_patch_absent.json --engine nnterp --device-map cuda \
    --data-root data --out out/absent
PlanError: write 'patch' has nothing to write on row(s) {1: 'alignment_missing'}. A read may skip a row; a write may not — writing nothing somewhere is not an intervention, and the row would score as if one had happened. `alignment_ambiguous` is a value that is in the prompt more than once, and scoping the anchor is what makes it one
$ echo $?
1

Through NDIF the same refusal arrives, with the same text and the same exit code, at the bottom of a 45-line double traceback:

$ NDIF_HOST=http://localhost:8001 causalab-mini run entity_patch_absent.json \
    --engine ndif --data-root data --out out/absent
Traceback (most recent call last):
  ...
  File "causalab_mini/cli.py", line 331, in run
    written = engine.execute(built, remote=remote, batch_size=args.batch_size).write(args.out)
  File "causalab_mini/engine/engines/nnterp/engine.py", line 89, in execute
    with model.session(remote=remote):
  File "nnsight/tracing/tracer.py", line 580, in __exit__
    raise exception.with_traceback(
nnsight.intervention.backends.remote.RemoteError: Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/ndif/services/ray/deployments/modeling/nns.py", line 183, in execute_traced_block
    tracer.execute(tracer.info.code)
  File "causalab_mini/engine/engines/nnterp/engine.py", line 96, in execute
    steps.run(engine, plan, batch_size=batch_size)
  File "/usr/local/lib/python3.12/site-packages/causalab_mini/engine/steps.py", line 76, in run
    run(engine, child, state.child() if isinstance(child, Plan) else state)
  ...
  File "/usr/local/lib/python3.12/site-packages/causalab_mini/engine/steps.py", line 481, in _writes_land
    raise PlanError(
causalab_mini.plan.plan.PlanError: write 'patch' has nothing to write on row(s) {1: 'alignment_missing'}. A read may skip a row; a write may not — writing nothing somewhere is not an intervention, and the row would score as if one had happened. `alignment_ambiguous` is a value that is in the prompt more than once, and scoping the anchor is what makes it one
$ echo $?
1

Three things to read off it. The message survives whole, down to the trailing advice, and the server's traceback comes with it. The frames say which copy ran: every mini frame below the block is the image's site-packages, and the one client-path frame is the traced block itself, which nnsight registers under the client's filename so the traceback can show the offending line. And the server knew what it was — its own log says error_type=PlanError:

ERROR [ndif.modeling] model execution errored … request_id=b645c48f52574292a39b1764223b7b23
  api_key=local stage=running error_type=PlanError fatal=False exec_ms=21.31

The other refusal worth knowing is the hooks engine's: it raises rather than run remote, because a hook is a Python callable on a module object in this process and there is nothing to ship. Remote is the nnterp engine's. --engine hooks never sets it, so you meet this one only from Python.

remote="local" #

execute takes remote as bool | str and passes it straight to the session, so remote="local" is nnsight's local simulation of a remote run: the session is serialized exactly as it would be for a server, this project's modules are hidden, and it is deserialized against the persistent objects a server supplies — which is how model.tokenizer resolves to the served checkpoint's. It is not reachable from the command line; the tests use it, and it costs no server:

$ uv run pytest tests/test_patching.py::test_remote_local_runs_the_same_plan_and_gets_the_same_numbers \
    tests/test_ragged.py::test_a_text_anchor_resolves_the_same_way_through_the_serialized_path -q
..                                                                       [100%]
2 passed, 1 warning in 13.79s

What it pins is the way out: a block that closed over something it should not, an anchor that resolved against the wrong tokenizer, a plan carrying something unpicklable.

What it does not cover is everything that is a fact about the other machine. It does not serialize the way home, so nothing it does proves a result can come back. It has one Python, one nnsight and one install, so no version skew can appear. It runs at the dtype you loaded, on the device you loaded, and not inside NDIF's autocast region. The first runs against a real server found seven defects invisible to the suite for exactly these reasons — among them that a filled-in plan cannot be pickled home at all, which is why execute brings back a flat {step path: {name: tensor}} and fills in the plan the client never gave up, and that NDIF runs a request under autocast, which downcast half of a Cayley solve until featurizer math was pinned outside it.

Running your own server #

Two stages: NDIF's own image, then the three checkouts on top of it, so the server can import what a block names. From a checkout of ndif:

V=$(git describe --tags | sed -E 's/^v//; s/-([0-9]+)-g/.post\1+g/')
docker build -f docker/Dockerfile --build-arg TORCH_CUDA=cu126 --build-arg NDIF_VERSION="$V" -t ndif-local:base .

Stage two installs nnsight, nnterp and causalab_mini with --no-deps — every dependency is pinned in the base image, and letting pip resolve here pulls a different torch — and passes each version in, because setuptools-scm cannot see a git history in a build context:

FROM ndif-local:base
COPY nnsight/ /src/nnsight/
RUN SETUPTOOLS_SCM_PRETEND_VERSION_FOR_NNSIGHT="0.8.1.dev125+g524c33fc" \
    pip install --no-deps --force-reinstall /src/nnsight \
 && python -c "import nnsight._c.py_mount"
COPY nnterp/ /src/nnterp/
RUN SETUPTOOLS_SCM_PRETEND_VERSION_FOR_NNTERP="0.5.0.dev0+internalsaccessors" \
    pip install --no-deps /src/nnterp
COPY causalab_mini_src/ /src/causalab_mini/
RUN pip install --no-deps /src/causalab_mini
RUN python -c "import nnsight, nnterp, causalab_mini, causalab_mini.engine.steps"

Build the context with git archive out of each checkout, so the image is a function of three commits and nothing else. Then run it and deploy the checkpoints the documents name:

docker run -d --name ndif-mini --gpus all --shm-size 4g -p 8001:8001 -p 9000:9000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface -e HF_TOKEN="$(cat ~/.cache/huggingface/token)" \
  ndif-local:mini

docker exec ndif-mini ndif deploy meta-llama/Llama-3.2-1B \
  --revision 4e20de362430cd3b72f300e6b0f18e50e7166e08 --dtype float32 --pinned

The HOT set lives in the controller's memory, so docker start brings the stack back without the deployments; re-run ndif deploy. The import nnsight._c.py_mount check in stage two is deliberate: without the C extension .save() fails on the server with an error that names neither.

The public service #

https://api.ndif.us is the default host, and the same /env call says what it has:

$ curl -s https://api.ndif.us/env | jq -r '.python_version, (.packages | {nnsight, nnterp, causalab_mini, torch, transformers} | to_entries[] | "\(.key) \(.value)")'
3.12.14 (main, Aug 14 2026, 15:34:45) [Clang 22.1.3 ]
nnsight 0.7.0
nnterp 1.3.0
causalab_mini null
torch 2.6.0+cu126
transformers 5.8.0

null is the answer: mini is not installed there, so a document cannot run on the public service — the block names modules the server cannot import. The nnsight line is a second reason, since a block from a 0.8 client is not something a 0.7 server can run. Remote means a server you or your group control, which is what the numbers on this page were measured against.