Welcome to TokSearch

TokSearch is a Python package for parallel retrieving, processing, and filtering of arbitrary-dimension fusion experimental data. TokSearch provides a high level API for extracting information from many shots, along with useful classes for low level data retrieval and manipulation.

The fundamental class in TokSearch is the Pipeline. A Pipeline object takes a list of shots and, for each shot in the list, creates a dict-like object called a Record. The Pipeline object then provides methods for defining a sequence of processing steps to apply to each record. These processing steps include:

Passing user-defined functions to the pipeline via the map method.

or...

Using a set of built-in methods, such as fetch, fetch_dataset, align, keep, or discard.

The Pipeline also provides a where method which takes as input a user-defined function that returns a boolean value. If the function evaluates to False for a record, then that record is removed from the pipeline.

Talk to your data

TokSearch packages its own know-how as agent-readable skills: how to build a pipeline, which signal class a quantity needs, how to align and aggregate. There are two ways to use them, and both are first-class. Pick your agent below, the built-in CLI or your own. Either way, run it through fdp.

The built-in conversational CLI

toksearch chat is a REPL with an LLM behind it. It writes the pipeline code, runs it against a persistent Python namespace, so follow-up turns iterate on cached results instead of re-fetching. It shows you each block before executing it.

toksearch chat                  # interactive REPL
toksearch chat --gui            # local Gradio GUI in a browser tab
toksearch query "..."           # one-shot, for scripts and quick lookups
toksearch backends              # list the names --backend accepts here

you> Use run_python to fetch ipmhd for shot 165920 from efit01.
[run_python] Fetch ipmhd via MdsSignal.
  pipeline = toksearch.Pipeline([165920])
  pipeline.fetch('ip', toksearch.MdsSignal(r'\ipmhd', 'efit01'))
  rec = list(pipeline.compute_serial())[0]
[output] (no output)

you> What's the peak |Ip| in MA?
[run_python] ...
[output] 1.1325

Backends: the Anthropic API, the OpenAI API, your Claude Max plan via the Claude Agent SDK, or the American Science Cloud (AmSC) endpoint contributed by toksearch_d3d.

Your own agent

If you already work in Claude Code, Cursor, or Codex, point it at the same material instead of switching tools. The skills install directly, and the same set is served over MCP by a standalone server:

fdp skills list                  # what's available, and what's installed
fdp skills install               # → ~/.claude/skills
fdp skills install --backend cursor    # or codex, or all; -f to overwrite

claude mcp add toksearch-skills -- fdp run python -m toksearch.llm.mcp

The server exposes every skill as a skill://<name> MCP resource plus a read_skill tool, so any MCP-capable client can browse and read them.

Preferred: run either path through `fdp`

Both paths are best used from an environment installed with the fdp-core metapackage (the standard FDP install; see Installation), through the fdp wrappers, which come with the device packages. fdp chat and fdp query configure data access (XRootD transport, MDSplus tree paths, bearer token) before the session starts, so the agent can actually reach shot data; fdp skills installs the skill files, and on the Your own agent path it is the fdp run in the claude mcp add line above that configures access.

fdp chat                        # interactive, FDP environment configured
fdp query "Fetch ip for shot 200000 and report the peak in MA."

See the LLM tutorial for an end-to-end walkthrough and the LLM Interface reference for the full API surface.

Installation

Recommended: the full FDP environment

Nearly everyone should install TokSearch as part of the Fusion Data Platform, through the fdp-core metapackage. That gets you TokSearch together with the device packages (toksearch_d3d, toksearch_mast), ptdata, imas_composer, the fdp CLI, the XRootD/Pelican transport that reaches DIII-D data, and the toksearch_cmf provenance backend, all pinned to a tested, mutually compatible set.

conda create -n fdp-installer -c ga-fdp -c conda-forge fdp-installer
conda activate fdp-installer
fdp-install -d /path/to/project
cd /path/to/project
pixi shell

fdp-install writes a pixi.toml that depends on fdp-core and runs pixi install. Substitute mamba or micromamba for conda if you prefer; the -n name is arbitrary.

Flag	Effect
`-d`, `--directory`	Where to install (default: the current directory)
`--latest`	Depend on `fdp-core-latest` (`>=` floors, e.g. `toksearch >=2.11.1`) instead of `fdp-core` (exact `==` pins, e.g. `toksearch ==2.11.1`)
`--install-skills`	Also install the agent skills to `~/.claude/skills/`

Inside that environment, run your analysis through the fdp CLI so the data-access environment is configured:

fdp run python my_pipeline.py
fdp run jupyter lab

TokSearch on its own

If you want the framework without FDP data access, for your own MDSplus server or Zarr stores or just the Pipeline machinery, install the package by itself:

conda install -c ga-fdp -c conda-forge toksearch

or into a fresh environment:

mamba create -n toksearch -c ga-fdp -c conda-forge toksearch

This does not give you DIII-D access: no toksearch_d3d, no fdp CLI, no XRootD/Pelican transport.

TokSearch is not published on PyPI. We intend to provide a pip install in the future.

From source

The repository is managed with pixi:

git clone https://github.com/GA-FDP/toksearch.git
cd toksearch
pixi install
pixi run build          # editable install into the pixi environment
pixi run test           # run the test suite

To build the documentation site locally:

pixi run -e docs docs-serve