Skip to content

Provenance and CMF

To reuse a result you need to know what produced it. TokSearch works that out for you. For every pipeline run it derives the shot source, the specification of each signal fetched, the sequence of operations, the compute backend, and the git commit of the script that ran, then hands all of it to a provenance backend. There is no metadata to write by hand.

Recording a run

Pass a backend to any of TokSearch's compute_* methods. JsonProvenance needs no external services and is the easiest way to see what gets captured:

from toksearch import Pipeline
from toksearch.provenance import JsonProvenance

run = JsonProvenance("demo", stage="explore", path="run.json")
results = pipeline.compute_multiprocessing(num_workers=16, provenance=run)
run.metrics("coverage", {"requested": len(shots), "returned": len(results)})
run.finalize()

The record identifies the inputs separately from the processing, so two runs that read the same data share an input identity even when they analyze it differently.

📚 Full reference, including Pipeline.write and the hook semantics: 👉 TokSearch provenance documentation

Recording to CMF

The Common Metadata Framework (CMF) treats code, data, and metadata as a single versioned unit, tracking the full graph of inputs, transformations, and outputs. It builds on Git, DVC, and MLMD.

The toksearch_cmf package connects the two: its CmfRun is a provenance backend that records a TokSearch run into CMF. It owns every cmflib and dvc dependency, so TokSearch core knows about neither.

from toksearch_cmf import CmfRun

run = CmfRun("betan-ip-study", stage="assemble", work_dir=".")
results = pipeline.compute_multiprocessing(num_workers=8, provenance=run)
run.finalize()

Both cmflib and toksearch_cmf are members of fdp-core, so an FDP environment already has them. There is no opt-in flag.

A CMF-tracked workflow: load and preprocess, label, train and evaluate, with each step's artifacts and metadata recorded alongside it. Metadata such as the signals used, shots used and archive version is captured per artifact, and successive commits (v0, v1) extend the graph rather than replacing it.

The metadata in that diagram is what gets recorded. The signals used, the shots used and the archive version all come from RunContext, which TokSearch derives from your pipeline.

Prerequisites. cmflib records the executing script's commit and hands output paths to DVC, so the script must run from inside a git repository with a remote and DVC initialised, with the script itself committed there. CmfRun checks for the repository up front rather than failing after a long compute.

A complete worked example against real DIII-D data is examples/betan_ip_peaks_cmf.py.

Why it is built this way

TokSearch core defines only an interface: a RunContext that describes the run, and a Provenance backend that consumes it. Recording to CMF is one implementation of that interface and writing JSON is another. A site that needs to record somewhere else can add its own without touching any pipeline code.

The two hooks TokSearch invokes itself are failure-isolated, so a broken provenance backend warns rather than killing a compute that has already finished.

Sharing results

CMF is designed so that curated outputs can be pushed from a local repository to a shared one, carrying their lineage with them, and so that its metadata can map onto community standards like IMAS. A searchable portal over those shared repositories is not yet built.

📚 Upstream CMF documentation: 👉 https://hewlettpackard.github.io/cmf/