edb

engineering_database.py

Module for dealing with JWST DMS Engineering Database mnemonics.

This module provides jwql with convenience classes and functions to retrieve and manipulate mnemonics from the JWST DMS EDB. It uses the engdb_tools module of the jwst package to interface the EDB directly.

Authors

• Johannes Sahlmann

• Mees Fix

• Bryan Hilbert

Use

This module can be imported and used with

from jwql.edb.engineering_database import get_mnemonic
get_mnemonic(mnemonic_identifier, start_time, end_time)

Required arguments:

mnemonic_identifier - String representation of a mnemonic name. start_time - astropy.time.Time instance end_time - astropy.time.Time instance

Notes

There are two possibilities for MAST authentication:

1. A valid MAST authentication token is present in the local jwql configuration file (config.json). 2. The MAST_API_TOKEN environment variable is set to a valid MAST authentication token.

When querying mnemonic values, the underlying MAST service returns data that include the datapoint preceding the requested start time and the datapoint that follows the requested end time.

class jwql.edb.engineering_database.EdbMnemonic(mnemonic_identifier, start_time, end_time, data, meta, info, blocks=[None], mean_time_block=None)

Class to hold and manipulate results of DMS EngDB queries.

block_stats(sigma=3, ignore_vals=[], ignore_edges=False, every_change=False)

Calculate stats for a mnemonic where we want a mean value for each block of good data, where blocks are separated by times where the data are ignored.

Parameters:

sigmaint

Number of sigma to use for sigma clipping

ignore_valslist

Any elements with values matching values in this list will be ignored

ignore_edgesbool

If True, the first and last elements of each block will be ignored. This is intended primarily for the MIRI ever_change data in IMIR_HK_xxx_POS_RATIO, where the position ratio values are not exactly synced up with the IMIR_HK_xxx_CUR_POS value. In that case, the first or last elements can have values from a time when the ratio has not yet settled to its final value.

every_changebool

If True, the data are assumed to be every_change data. This is used when dealing with blocks that exclusively contain data to be ignored

block_stats_filter_positions(sigma=5)

Calculate stats for a mnemonic where we want a mean value for each block of good data, where blocks are separated by times where the data are ignored. In this case, there are custom adjustments meant to work on the MIRI filter position mnemonics (e.g. IMIR_HK_GW14_POS_RATIO, IMIR_HK_FW_POS_RATIO).

Parameters:

sigmaint

Number of sigma to use for sigma clipping

bokeh_plot(show_plot=False, savefig=False, out_dir='./', nominal_value=None, yellow_limits=None, red_limits=None, title=None, xrange=(None, None), yrange=(None, None), return_components=True, return_fig=False, plot_data=True, plot_mean=False, plot_median=False, plot_max=False, plot_min=False)

Make basic bokeh plot showing value as a function of time. Optionally add a line indicating nominal (expected) value, as well as yellow and red background regions to denote values that may be unexpected.

Parameters:

show_plotbool

If True, show plot on screen rather than returning div and script

savefigbool

If True, file is saved to html file

out_dirstr

Directory into which the html file is saved

nominal_valuefloat

Expected or nominal value for the telemetry. If provided, a horizontal dashed line at this value will be added.

yellow_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered non-nominal. If provided, the area of the plot between these two values will be given a green background, and that outside of these limits will have a yellow background.

red_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered worse than in the yellow region. If provided, the area of the plot outside of these two values will have a red background.

titlestr

Will be used as the plot title. If None, the mnemonic name and description (if present) will be used as the title

xrangetuple

Tuple of min, max datetime values to use as the plot range in the x direction.

yrangetuple

Tuple of min, max datetime values to use as the plot range in the y direction.

return_componentsbool

If True, return the plot as div and script components

return_figbool

If True, return the plot as a bokeh Figure object

plot_databool

If True, plot the data in the EdbMnemonic.data table

plot_meanbool

If True, also plot the line showing the self.mean values

plot_medianbool

If True, also plot the line showing the self.median values

plot_maxbool

If True, also plot the line showing the self.max values

plot_minbool

If True, also plot the line showing the self.min values

Returns:

objlist or bokeh.plotting.figure

If return_components is True, return a list containing [div, script] If return_figre is True, return the bokeh figure itself

bokeh_plot_text_data(show_plot=False)

Make basic bokeh plot showing value as a function of time.

Parameters:

show_plotboolean

A switch to show the plot in the browser or not.

Returns:

[div, script]list

List containing the div and js representations of figure.

change_only_add_points()

Tweak change-only data. Add an additional data point immediately prior to each original data point, with a value equal to that in the previous data point. This will help with filtering data based on conditions later, and will create a plot that looks more realistic, with only horizontal and vertical lines.

daily_stats(sigma=3)

Calculate the statistics for each day in the data contained in data[“data”]. Should we add a check for a case where the final block of time is <<1 day?

Parameters:

sigmaint

Number of sigma to use for sigma clipping

full_stats(sigma=3)

Calculate the mean/median/stdev of the full compliment of data

Parameters:

sigmaint

Number of sigma to use for sigma clipping

get_table_data()

Get data needed to make interactive table in template.

interpolate(times)

Interpolate data euvalues at specified datetimes.

Parameters:

timeslist

List of datetime objects describing the times to interpolate to

plot_data_plus_devs(use_median=False, show_plot=False, savefig=False, out_dir='./', nominal_value=None, yellow_limits=None, red_limits=None, xrange=(None, None), yrange=(None, None), title=None, return_components=True, return_fig=False, plot_max=False, plot_min=False)

Make basic bokeh plot showing value as a function of time. Optionally add a line indicating nominal (expected) value, as well as yellow and red background regions to denote values that may be unexpected. Also add a plot of the mean value over time and in a second figure, a plot of the devaition from the mean.

Parameters:

use_medianbool

If True, plot the median rather than the mean, as well as the deviation from the median rather than from the mean

show_plotbool

If True, show plot on screen rather than returning div and script

savefigbool

If True, file is saved to html file

out_dirstr

Directory into which the html file is saved

nominal_valuefloat

Expected or nominal value for the telemetry. If provided, a horizontal dashed line at this value will be added.

yellow_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered non-nominal. If provided, the area of the plot between these two values will be given a green background, and that outside of these limits will have a yellow background.

red_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered worse than in the yellow region. If provided, the area of the plot outside of these two values will have a red background.

xrangetuple

Tuple of min, max datetime values to use as the plot range in the x direction.

yrangetuple

Tuple of min, max datetime values to use as the plot range in the y direction.

titlestr

Will be used as the plot title. If None, the mnemonic name and description (if present) will be used as the title

return_componentsbool

If True, return the plot as div and script components

return_figbool

If True, return the plot as a bokeh Figure object

plot_maxbool

If True, also plot the line showing the self.max values

plot_minbool

If True, also plot the line showing the self.min values

Returns:

objlist or bokeh.plotting.figure

If return_components is True, return a list containing [div, script] If return_figre is True, return the bokeh figure itself

save_table(outname)

Save the EdbMnemonic instance

Parameters:

outnamestr

Name of text file to save information into

timed_stats(sigma=3)

Break up the data into chunks of the given duration. Calculate the mean value for each chunk.

Parameters:

sigmaint

Number of sigma to use in sigma-clipping

jwql.edb.engineering_database.add_limit_boxes(fig, yellow=None, red=None)

Add green/yellow/red background colors

Parameters:

figbokeh.plotting.figure

Bokeh figure of the telemetry values

yellowlist

2-element list of [low, high] values. If provided, the areas of the plot less than <low> and greater than <high> will be given a yellow background, to indicate an area of concern.

redlist

2-element list of [low, high] values. If provided, the areas of the plot less than <low> and greater than <high> will be given a red background, to indicate values that may indicate an error. It is assumed that the low value of red is less than the low value of yellow, and that the high value of red is greater than the high value of yellow.

Returns:

figbokeh.plotting.figure

Modified figure with BoxAnnotations added

jwql.edb.engineering_database.add_time_offset(offset, dt_obj)

Add an offset to an input datetime object

Parameters:

offsetfloat

Number of seconds to be added

dt_objdatetime.datetime

Datetime object to which the seconds are added

Returns:

objdatetime.datetime

Sum of the input datetime objects and the offset seconds.

jwql.edb.engineering_database.calc_median_time(time_arr)

Calcualte the median time of the input time_arr

Parameters:

time_arrnumpy.ndarray

1D array of datetime objects

Returns:

med_timedatetime.datetime

Median time, as a datetime object

jwql.edb.engineering_database.change_only_bounding_points(date_list, value_list, starttime, endtime)

For data containing change-only values, where bracketing data outside the requested time span may be present, create data points at the starting and ending times. This can be helpful with later interpolations.

Parameters:

date_listlist

List of datetime values

value_listlist

List of corresponding mnemonic values

starttimedatetime.datetime

Start time

endtimedatetime.datetime

End time

Returns:

date_listlist

List of datetime values

value_listlist

List of corresponding mnemonic values

jwql.edb.engineering_database.change_only_stats(times, values, sigma=3)

Calculate the mean/median/stdev as well as the median time for a collection of change-only data.

Parameters:

timeslist

List of datetime objects

valueslist

List of values corresponding to times

sigmafloat

Number of sigma to use for sigma-clipping

Returns:

meanvalfloat

Mean of values

medianvalfloat

Median of values

stdevvalfloat

Standard deviation of values

jwql.edb.engineering_database.create_time_offset(dt_obj, epoch)

Subtract input epoch from a datetime object and return the residual number of seconds

Parameters:

dt_objdatetime.datetime

Original datetiem object

epochdatetime.datetime

Datetime to be subtracted from dt_obj

Returns:

objfloat

Number of seconds between dt_obj and epoch

jwql.edb.engineering_database.get_mnemonic(mnemonic_identifier, start_time, end_time)

Execute query and return an EdbMnemonic instance.

The underlying MAST service returns data that include the datapoint preceding the requested start time and the datapoint that follows the requested end time.

Parameters:

mnemonic_identifierstr

Telemetry mnemonic identifiers, e.g. SA_ZFGOUTFOV

start_timeastropy.time.Time or datetime.datetime

Start time

end_timeastropy.time.Time or datetime.datetime

End time

Returns:

mnemonicinstance of EdbMnemonic

EdbMnemonic object containing query results

jwql.edb.engineering_database.get_mnemonic_info(mnemonic_identifier)

Return the mnemonic description.

Parameters:

mnemonic_identifierstr

Telemetry mnemonic identifier, e.g. SA_ZFGOUTFOV

Returns:

infodict

Object that contains the returned data

jwql.edb.engineering_database.get_mnemonics(mnemonics, start_time, end_time)

Query DMS EDB with a list of mnemonics and a time interval.

Parameters:

mnemonicslist or numpy.ndarray

Telemetry mnemonic identifiers, e.g. ['SA_ZFGOUTFOV', 'IMIR_HK_ICE_SEC_VOLT4']

start_timeastropy.time.Time instance

Start time

end_timeastropy.time.Time instance

End time

Returns:

mnemonic_dictdict

Dictionary. keys are the queried mnemonics, values are instances of EdbMnemonic

jwql.edb.engineering_database.interpolate_datetimes(new_times, old_times, old_data)

interpolate a set of datetime/value pairs onto another set of datetime objects

Parameters:

new_timesnumpy.ndarray

Array of datetime objects onto which the data will be interpolated

old_timesnumpy.ndarray

Array of datetime objects associated with the input data values

old_datanumpy.ndarray

Array of data values associated with old_times, which will be interpolated onto new_times

Returns:

new_datanumpy.ndarray

Array of values interpolated onto new_times

jwql.edb.engineering_database.is_valid_mnemonic(mnemonic_identifier)

Determine if the given string is a valid EDB mnemonic.

Parameters:

mnemonic_identifierstr

The mnemonic_identifier string to be examined.

Returns:

bool

Is mnemonic_identifier a valid EDB mnemonic?

jwql.edb.engineering_database.mnemonic_inventory()

Return all mnemonics in the DMS engineering database. No authentication is required, this information is public. Since this is a rather large and quasi-static table (~15000 rows), it is cached using functools.

Returns:

dataastropy.table.Table

Table representation of the mnemonic inventory.

metadict

Additional information returned by the query.

jwql.edb.engineering_database.process_mast_service_request_result(result, data_as_table=True)

Parse the result of a MAST EDB query.

Parameters:

resultlist of requests.models.Response instances

The object returned by a call to Mast.service_request_async

data_as_tablebool

If True, return data as astropy table, else return as json

Returns:

dataastropy.table.Table

Table representation of the returned data.

metadict

Additional information returned by the query

jwql.edb.engineering_database.query_mnemonic_info(mnemonic_identifier, token=None)

Query the EDB to return the mnemonic description.

Parameters:

mnemonic_identifierstr

Telemetry mnemonic identifier, e.g. SA_ZFGOUTFOV

tokenstr

MAST token

Returns:

infodict

Object that contains the returned data