Ray

toksearch.backend.ray.RayConfig dataclass

Configuration for the Ray backend

Parameters:

Name	Type	Description	Default
numparts	Optional[int]	The number of partitions to use when mapping. If not provided, the number of partitions will equal the number of records in the pipeline.	None
batch_size	Optional[int]	The number of elements to process in each batch. Defaults to the number of records in the pipeline.	None
verbose	bool	Whether to print verbose output. Default is True.	True
placement_group_func	Optional[Callable]	A function that returns a placement group. See the ray docs for more information on placement groups.	None
memory_per_task	Optional[int]	Memory to allocate to each task in bytes. If not provided, there is no limit.	None
ray_init_kwargs	Optional[dict]	Dict of keyword arguments to pass to ray.init	None

toksearch.backend.ray.RayRecordSet

Bases: RecordSet

__init__(raydd)

Create a RayRecordSet from a RayDD.

cleanup()

Shutdown the Ray cluster.

All Ray objects will be lost after this method is called, so only call it after all records have been copied back to the local machine, either by grabbing an index, or slice of the RayRecordSet or converting it to a list.

from_records(records, config=None) classmethod

Create a RayRecordSet from a list of records.

Parameters:

Name	Type	Description	Default
records	List[Record]	List of records to create the RecordSet from.	required
config	Optional[RayConfig]	Configuration for the Ray backend.	None