Ray
toksearch.backend.ray.RayConfig
dataclass
Configuration for the Ray backend
Parameters:
Name | Type | Description | Default |
---|---|---|---|
numparts |
Optional[int]
|
The number of partitions to use when mapping. If not provided, the number of partitions will equal the number of records in the pipeline. |
None
|
batch_size |
Optional[int]
|
The number of elements to process in each batch. Defaults to the number of records in the pipeline. |
None
|
verbose |
bool
|
Whether to print verbose output. Default is True. |
True
|
placement_group_func |
Optional[Callable]
|
A function that returns a placement group. See the ray docs for more information on placement groups. |
None
|
memory_per_task |
Optional[int]
|
Memory to allocate to each task in bytes. If not provided, there is no limit. |
None
|
ray_init_kwargs |
Optional[dict]
|
Dict of keyword arguments to pass to ray.init |
None
|
toksearch.backend.ray.RayRecordSet
Bases: RecordSet
__init__(raydd)
Create a RayRecordSet from a RayDD.
cleanup()
Shutdown the Ray cluster.
All Ray objects will be lost after this method is called, so only call it after all records have been copied back to the local machine, either by grabbing an index, or slice of the RayRecordSet or converting it to a list.