Working with signals¶
The Signal class and its derived subclasses provide methods for fetching data from some external source. At present, this includes MDSplus archives (with the ability to access both tree data on the local file system and remote servers).
Note: Access to the DIII-D PTDATA archiving system is available in the PtDataSignal class, which is provided by the toksearch_d3d package, which must be installed separately. Examples are provided in the toksearch_d3d package documentation.
Behind the scenes, things like connection handles and open file descriptors are cached. Signal objects provide methods for cleaning up those resources both after processing a single shot (such as with local MDSplus tree files) and after running many shots (such as persistent network connections). When used in conjunction with a Pipeline, the Pipeline object will handle the resource management aspect.
Creating an MdsSignal¶
As an example of using a Signal object, we'll look at the MdsSignal, which is built into the core TokSearch package.
import numpy as np
import pprint
from toksearch import MdsSignal
np.set_printoptions(threshold=3, precision=1)
mds_node = r'\ipmhd'
mds_tree = 'efit01'
location = None
ipmhd_signal = MdsSignal(mds_node, mds_tree, location=location)
- We set the first argument specifying which MDSplus data we want, in this case the reconstructed plasma current,
r'\ipmhd. Any valid mds node or tdi expression is allowed. Note that we use the Python raw specifier,r, to ensure that the\character is not interpreted as an escape character. - We specify the MDSplus tree as
efit01. This documentation was developed using data from DIII-D, so this is data from a DIII-D equilibrium reconstruction. - We'll generally NOT specify an explicit
location, as this will usually taken care of by the defaults in the environment. When location is not specified, the MdsSignal will first look for an environment variable called TOKSEARCH_MDS_DEFAULT and use that if found. Otherwise, it will look for an environment variable of the form TREE_path (which in this case would be efit01_path). Failing that, it will raise an exception. To specify that we're using a remote server, we would instead use the syntaxlocation="remote://servername". For example, at DIII-D, we might uselocation="remote://atlas.gat.com".
Every Signal subclass provides methods for fetching data. When used with a Pipeline, these would not be called directly. But, they provide a convenient interface with the archives and are also useful for debugging.
shot = 165920
# Fetch the data into a dict with fields data and times
dict_result = ipmhd_signal.fetch(shot)
print(dict_result)
{'data': array([213493.6, 281801.6, 286739.6, ..., 475302.8, 474772.5, 471811.7],
dtype=float32), 'times': array([ 100., 140., 160., ..., 6340., 6360., 6380.], dtype=float32), 'units': {'data': 'A', 'times': 'ms'}}
# We can also fetch the result as an xarray DataArray object
xarray_result = ipmhd_signal.fetch_as_xarray(shot)
print(xarray_result)
<xarray.DataArray (times: 303)> Size: 1kB
array([213493.6, 281801.6, 286739.6, ..., 475302.8, 474772.5, 471811.7],
dtype=float32)
Coordinates:
* times (times) float32 1kB 100.0 140.0 160.0 ... 6.36e+03 6.38e+03
Attributes:
units: A
Signal Callbacks¶
All Signal classes allow the user to provide a callback for modifying both the fetched data and coordinates associated with its dimensions. These callbacks are supplied by running the set_callback method, and then are executed during execution of either the fetch or fetch_as_xarray methods.
The example below shows a simple example of a callback. It simply does unit conversion for the ip data from MA to kA, and the timebase from milliseconds to seconds.
def MA_to_kA(result_dict):
result_dict["data"] *= 1000.
result_dict["times"] *= 0.001
result_dict["units"]["data"] = "kA"
result_dict["units"]["times"] = "s"
return result_dict
ip_signal_ka = MdsSignal(r"\ipmeas", "efit01").set_callback(MA_to_kA)
print(ip_signal_ka.fetch_as_xarray(shot))
<xarray.DataArray (times: 303)> Size: 1kB
array([2.1e+08, 2.7e+08, 2.8e+08, ..., 4.8e+08, 4.8e+08, 4.7e+08],
dtype=float32)
Coordinates:
* times (times) float32 1kB 0.1 0.14 0.16 0.18 0.2 ... 6.32 6.34 6.36 6.38
Attributes:
units: kA
Multidimensional Data¶
Signal classes support retrieval of data with more dimensions than just time. As an example, we'll look at retrieval of flux-on-the-grid, psirz, which has three dimension: times, r, and z. But we'll assume that we don't know that there are three dimensions, or what order they are stored in (since MDSplus doesn't really provide a easy way to figure this out).
We start by retrieving just the data and no dimensions. We need to do a bit of detective work to determine which dimensions are which.
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=None)
shape = psirz_sig.fetch(shot)['data'].shape
print(shape)
(303, 65, 65)
Now we know that there are three dimensions, but we don't know which dimension is which.
We pass the keyword argument dims a list of dummy labels.
dummy_dims = ['dim0', 'dim1', 'dim2']
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dummy_dims)
res = psirz_sig.fetch(shot)
for dim in dummy_dims:
print('{} shape: {}'.format(dim, res[dim].shape))
dim0 shape: (65,) dim1 shape: (65,) dim2 shape: (303,)
From this, we gather that the last dimension is the times dimension and that the other two dimensions represent r and z, in some order.
print('Min dim0: {}, Max dim0: {}'.format(res['dim0'].min(), res['dim0'].max()))
print('Min dim1: {}, Max dim1: {}'.format(res['dim1'].min(), res['dim1'].max()))
Min dim0: 0.8399999737739563, Max dim0: 2.5399999618530273 Min dim1: -1.600000023841858, Max dim1: 1.600000023841858
By convention, we know that r is always positive, while z can assume both negative, so we know we can label the dimensions as ['r', 'z', 'times'].
But, we now have another problem. The shape of the data is (303, 65, 65), which is inconsistent with the ordering of the dimensions. This is ok if use the fetch method:
dims = ['r', 'z', 'times']
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dims)
dict_result = psirz_sig.fetch(shot)
print('times: {}'.format(dict_result['times']))
print('r : {}'.format(dict_result['r']))
print('z : {}'.format(dict_result['z']))
times: [ 100. 140. 160. ... 6340. 6360. 6380.] r : [0.8 0.9 0.9 ... 2.5 2.5 2.5] z : [-1.6 -1.6 -1.5 ... 1.5 1.6 1.6]
However, running fetch_as_xarray requires that the dimension ordering be consistent.
try:
xarray_result = psirz_sig.fetch_as_xarray(shot)
except Exception as e:
print('ERROR: {}'.format(e))
ERROR: conflicting sizes for dimension 'r': length 303 on the data but length 65 on coordinate 'r'
We can fix this by using a data_func callback and performing a transpose operation on the data.
psirz_sig = MdsSignal(r'\psirz', 'efit01', dims=dims, data_order=['times', 'r', 'z'])
print(psirz_sig.fetch_as_xarray(shot))
<xarray.DataArray (times: 303, r: 65, z: 65)> Size: 5MB
array([[[-2.9e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.4e-01],
[-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.5e-01],
[-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.5e-01],
...,
[-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.5e-01],
[-3.0e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.5e-01],
[-2.9e-01, -3.0e-01, -3.0e-01, ..., -2.5e-01, -2.5e-01,
-2.4e-01]],
[[-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01,
-2.2e-01],
[-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01,
-2.3e-01],
[-2.9e-01, -2.9e-01, -2.9e-01, ..., -2.3e-01, -2.3e-01,
-2.3e-01],
...
[-3.0e-02, -3.1e-02, -3.1e-02, ..., 1.4e-03, 2.4e-03,
3.4e-03],
[-3.0e-02, -3.0e-02, -3.1e-02, ..., 5.4e-04, 1.5e-03,
2.4e-03],
[-2.9e-02, -3.0e-02, -3.0e-02, ..., -1.9e-04, 7.0e-04,
1.6e-03]],
[[-3.2e-02, -3.3e-02, -3.4e-02, ..., 3.5e-03, 4.5e-03,
5.4e-03],
[-3.3e-02, -3.4e-02, -3.5e-02, ..., 5.0e-03, 6.0e-03,
7.0e-03],
[-3.4e-02, -3.5e-02, -3.6e-02, ..., 6.8e-03, 7.9e-03,
8.9e-03],
...,
[-3.0e-02, -3.1e-02, -3.1e-02, ..., 1.2e-03, 2.2e-03,
3.2e-03],
[-3.0e-02, -3.0e-02, -3.1e-02, ..., 3.9e-04, 1.3e-03,
2.2e-03],
[-2.9e-02, -2.9e-02, -3.0e-02, ..., -3.4e-04, 5.4e-04,
1.4e-03]]], dtype=float32)
Coordinates:
* times (times) float32 1kB 100.0 140.0 160.0 ... 6.36e+03 6.38e+03
* r (r) float32 260B 0.84 0.8666 0.8931 0.9197 ... 2.487 2.513 2.54
* z (z) float32 260B -1.6 -1.55 -1.5 -1.45 -1.4 ... 1.45 1.5 1.55 1.6
Attributes:
units: V s / rad
Now everything works.