Working with signals¶
The Signal
class and its derived subclasses provide methods for fetching data from some external source. At present, this includes MDSplus archives (with the ability to access both tree data on the local file system and remote servers).
Note: Access to the DIII-D PTDATA archiving system is available in the PtDataSignal class, which is provided by the toksearch_d3d
package, which must be installed separately. Examples are provided in the toksearch_d3d
package documentation.
Behind the scenes, things like connection handles and open file descriptors are cached. Signal
objects provide methods for cleaning up those resources both after processing a single shot (such as with local MDSplus tree files) and after running many shots (such as persistent network connections). When used in conjunction with a Pipeline
, the Pipeline
object will handle the resource management aspect.
Creating an MdsSignal
¶
As an example of using a Signal
object, we'll look at the MdsSignal
, which is built into the core TokSearch package.
import numpy as np
import pprint
from toksearch import MdsSignal
np.set_printoptions(threshold=3, precision=1)
mds_node = r'\ipmhd'
mds_tree = 'efit01'
location = None
ipmhd_signal = MdsSignal(mds_node, mds_tree, location=location)
- We set the first argument specifying which MDSplus data we want, in this case the reconstructed plasma current,
r'\ipmhd
. Any valid mds node or tdi expression is allowed. Note that we use the Python raw specifier,r
, to ensure that the\
character is not interpreted as an escape character. - We specify the MDSplus tree as
efit01
. This documentation was developed using data from DIII-D, so this is data from a DIII-D equilibrium reconstruction. - We'll generally NOT specify an explicit
location
, as this will usually taken care of by the defaults in the environment. When location is not specified, the MdsSignal will first look for an environment variable called TOKSEARCH_MDS_DEFAULT and use that if found. Otherwise, it will look for an environment variable of the form TREE_path (which in this case would be efit01_path). Failing that, it will raise an exception. To specify that we're using a remote server, we would instead use the syntaxlocation="remote://servername"
. For example, at DIII-D, we might uselocation="remote://atlas.gat.com"
.
Every Signal
subclass provides methods for fetching data. When used with a Pipeline
, these would not be called directly. But, they provide a convenient interface with the archives and are also useful for debugging.
shot = 165920
# Fetch the data into a dict with fields data and times
dict_result = ipmhd_signal.fetch(shot)
print(dict_result)
{'data': array([213493.6, 281801.6, 286739.6, ..., 475302.8, 474772.5, 471811.7], dtype=float32), 'times': array([ 100., 140., 160., ..., 6340., 6360., 6380.], dtype=float32), 'units': {'data': 'A', 'times': 'ms'}}
# We can also fetch the result as an xarray DataArray object
xarray_result = ipmhd_signal.fetch_as_xarray(shot)
print(xarray_result)
<xarray.DataArray (times: 303)> Size: 1kB array([213493.6, 281801.6, 286739.6, ..., 475302.8, 474772.5, 471811.7], dtype=float32) Coordinates: * times (times) float32 1kB 100.0 140.0 160.0 ... 6.36e+03 6.38e+03 Attributes: units: A
Signal
Callbacks¶
All Signal
classes allow the user to provide a callback for modifying both the fetched data and coordinates associated with its dimensions. These callbacks are supplied by running the set_callback
method, and then are executed during execution of either the fetch
or fetch_as_xarray
methods.
The example below shows a simple example of a callback. It simply does unit conversion for the ip
data from MA to kA, and the timebase from milliseconds to seconds.
def MA_to_kA(result_dict):
result_dict["data"] *= 1000.
result_dict["times"] *= 0.001
result_dict["units"]["data"] = "kA"
result_dict["units"]["times"] = "s"
return result_dict
ip_signal_ka = MdsSignal(r"\ipmeas", "efit01").set_callback(MA_to_kA)
print(ip_signal_ka.fetch_as_xarray(shot))
<xarray.DataArray (times: 303)> Size: 1kB array([2.1e+08, 2.7e+08, 2.8e+08, ..., 4.8e+08, 4.8e+08, 4.7e+08], dtype=float32) Coordinates: * times (times) float32 1kB 0.1 0.14 0.16 0.18 0.2 ... 6.32 6.34 6.36 6.38 Attributes: units: kA
Multidimensional Data¶
Signal
classes support retrieval of data with more dimensions than just time. As an example, we'll look at retrieval of flux-on-the-grid, psirz
, which has three dimension: times, r, and z. But we'll assume that we don't know that there are three dimensions, or what order they are stored in (since MDSplus doesn't really provide a easy way to figure this out).
We start by retrieving just the data and no dimensions. We need to do a bit of detective work to determine which dimensions are which.
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=None)
shape = psirz_sig.fetch(shot)['data'].shape
print(shape)
(303, 65, 65)
Now we know that there are three dimensions, but we don't know which dimension is which.
We pass the keyword argument dims
a list of dummy labels.
dummy_dims = ['dim0', 'dim1', 'dim2']
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dummy_dims)
res = psirz_sig.fetch(shot)
for dim in dummy_dims:
print('{} shape: {}'.format(dim, res[dim].shape))
dim0 shape: (65,) dim1 shape: (65,) dim2 shape: (303,)
From this, we gather that the last dimension is the times
dimension and that the other two dimensions represent r
and z
, in some order.
print('Min dim0: {}, Max dim0: {}'.format(res['dim0'].min(), res['dim0'].max()))
print('Min dim1: {}, Max dim1: {}'.format(res['dim1'].min(), res['dim1'].max()))
Min dim0: 0.8399999737739563, Max dim0: 2.5399999618530273 Min dim1: -1.600000023841858, Max dim1: 1.600000023841858
By convention, we know that r
is always positive, while z
can assume both negative, so we know we can label the dimensions as ['r', 'z', 'times']
.
But, we now have another problem. The shape of the data
is (303, 65, 65)
, which is inconsistent with the ordering of the dimensions. This is ok if use the fetch
method:
dims = ['r', 'z', 'times']
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dims)
dict_result = psirz_sig.fetch(shot)
print('times: {}'.format(dict_result['times']))
print('r : {}'.format(dict_result['r']))
print('z : {}'.format(dict_result['z']))
times: [ 100. 140. 160. ... 6340. 6360. 6380.] r : [0.8 0.9 0.9 ... 2.5 2.5 2.5] z : [-1.6 -1.6 -1.5 ... 1.5 1.6 1.6]
However, running fetch_as_xarray
requires that the dimension ordering be consistent.
try:
xarray_result = psirz_sig.fetch_as_xarray(shot)
except Exception as e:
print('ERROR: {}'.format(e))
ERROR: conflicting sizes for dimension 'r': length 303 on the data but length 65 on coordinate 'r'
We can fix this by using a data_func
callback and performing a transpose operation on the data
.
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dims, data_order=['times', 'r', 'z'])
print(psirz_sig.fetch_as_xarray(shot))
<xarray.DataArray (times: 303, r: 65, z: 65)> Size: 5MB array([[[-2.9e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.4e-01], [-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.5e-01], [-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.5e-01], ..., [-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.5e-01], [-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.5e-01], [-2.9e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01, -2.4e-01]], [[-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01, -2.2e-01], [-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01, -2.3e-01], [-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01, -2.3e-01], ... [-3.0e-02, -3.1e-02, -3.1e-02, ..., 1.4e-03, 2.4e-03, 3.4e-03], [-3.0e-02, -3.0e-02, -3.1e-02, ..., 5.4e-04, 1.5e-03, 2.4e-03], [-2.9e-02, -3.0e-02, -3.0e-02, ..., -1.9e-04, 7.0e-04, 1.6e-03]], [[-3.2e-02, -3.3e-02, -3.4e-02, ..., 3.5e-03, 4.5e-03, 5.4e-03], [-3.3e-02, -3.4e-02, -3.5e-02, ..., 5.0e-03, 6.0e-03, 7.0e-03], [-3.4e-02, -3.5e-02, -3.6e-02, ..., 6.8e-03, 7.9e-03, 8.9e-03], ..., [-3.0e-02, -3.1e-02, -3.1e-02, ..., 1.2e-03, 2.2e-03, 3.2e-03], [-3.0e-02, -3.0e-02, -3.1e-02, ..., 3.9e-04, 1.3e-03, 2.2e-03], [-2.9e-02, -2.9e-02, -3.0e-02, ..., -3.4e-04, 5.4e-04, 1.4e-03]]], dtype=float32) Coordinates: * times (times) float32 1kB 100.0 140.0 160.0 ... 6.36e+03 6.38e+03 * r (r) float32 260B 0.84 0.8666 0.8931 0.9197 ... 2.487 2.513 2.54 * z (z) float32 260B -1.6 -1.55 -1.5 -1.45 -1.4 ... 1.45 1.5 1.55 1.6 Attributes: units: V s / rad
Now everything works.