RecordSet

toksearch.record.record_set.RecordSet

Bases: ABC

Abstract base class for a set of Records

This class provides a common interface for a set of Records that are produced by the various execution backends.

Methods:

Name	Description
__len__	Returns the number of records in the set
__getitem__	Returns a record by index
__iter__	Iterates over the records in the set as a generator
map	Applies one or more functions to the records
cleanup	Cleans up any resources used by the RecordSet, such as shutting down a SparkContext or Ray cluster

to_dataframe(fields=None)

Return a pandas DataFrame with one row per record.

Intended for scalar summary results. Array-valued and dataset-valued fields belong in per-shot files written by Pipeline.write, not in a driver-side frame.

Rows are built from each record's own keys rather than one shared column list: a shot whose map function failed never gains the field its siblings have, and pandas unions the columns and fills NaN. That keeps a partial run visibly partial instead of silently smaller.

Parameters:

Name	Type	Description	Default
fields		Field names to include, in order. When omitted, every field except bookkeeping ones is included. shot is always the first column.	None

to_parquet(path, fields=None)

Write the record set to a parquet file and return its path.

If this record set came from a compute_* call with a provenance backend, the file is declared to that backend as an output artifact. The file is written first: a provenance failure must not cost the user their output.