edb¶
engineering_database.py¶
Module for dealing with JWST DMS Engineering Database mnemonics.
This module provides jwql
with convenience classes and functions
to retrieve and manipulate mnemonics from the JWST DMS EDB. It uses
the engdb_tools
module of the jwst
package to interface the
EDB directly.
Use¶
This module can be imported and used with
from jwql.edb.engineering_database import get_mnemonic get_mnemonic(mnemonic_identifier, start_time, end_time)Required arguments:
mnemonic_identifier
- String representation of a mnemonic name.start_time
- astropy.time.Time instanceend_time
- astropy.time.Time instance
Notes¶
There are two possibilities for MAST authentication:
1. A valid MAST authentication token is present in the local
jwql
configuration file (config.json). 2. The MAST_API_TOKEN environment variable is set to a valid MAST authentication token.When querying mnemonic values, the underlying MAST service returns data that include the datapoint preceding the requested start time and the datapoint that follows the requested end time.
- class jwql.edb.engineering_database.EdbMnemonic(mnemonic_identifier, start_time, end_time, data, meta, info, blocks=[None], mean_time_block=None)[source]¶
Class to hold and manipulate results of DMS EngDB queries.
- block_stats(sigma=3, ignore_vals=[], ignore_edges=False, every_change=False)[source]¶
Calculate stats for a mnemonic where we want a mean value for each block of good data, where blocks are separated by times where the data are ignored.
- Parameters:
- sigmaint
Number of sigma to use for sigma clipping
- ignore_valslist
Any elements with values matching values in this list will be ignored
- ignore_edgesbool
If True, the first and last elements of each block will be ignored. This is intended primarily for the MIRI ever_change data in IMIR_HK_xxx_POS_RATIO, where the position ratio values are not exactly synced up with the IMIR_HK_xxx_CUR_POS value. In that case, the first or last elements can have values from a time when the ratio has not yet settled to its final value.
- every_changebool
If True, the data are assumed to be every_change data. This is used when dealing with blocks that exclusively contain data to be ignored
- block_stats_filter_positions(sigma=5)[source]¶
Calculate stats for a mnemonic where we want a mean value for each block of good data, where blocks are separated by times where the data are ignored. In this case, there are custom adjustments meant to work on the MIRI filter position mnemonics (e.g. IMIR_HK_GW14_POS_RATIO, IMIR_HK_FW_POS_RATIO).
- Parameters:
- sigmaint
Number of sigma to use for sigma clipping
- bokeh_plot(show_plot=False, savefig=False, out_dir='./', nominal_value=None, yellow_limits=None, red_limits=None, title=None, xrange=(None, None), yrange=(None, None), return_components=True, return_fig=False, plot_data=True, plot_mean=False, plot_median=False, plot_max=False, plot_min=False)[source]¶
Make basic bokeh plot showing value as a function of time. Optionally add a line indicating nominal (expected) value, as well as yellow and red background regions to denote values that may be unexpected.
- Parameters:
- show_plotbool
If True, show plot on screen rather than returning div and script
- savefigbool
If True, file is saved to html file
- out_dirstr
Directory into which the html file is saved
- nominal_valuefloat
Expected or nominal value for the telemetry. If provided, a horizontal dashed line at this value will be added.
- yellow_limitslist
2-element list giving the lower and upper limits outside of which the telemetry value is considered non-nominal. If provided, the area of the plot between these two values will be given a green background, and that outside of these limits will have a yellow background.
- red_limitslist
2-element list giving the lower and upper limits outside of which the telemetry value is considered worse than in the yellow region. If provided, the area of the plot outside of these two values will have a red background.
- titlestr
Will be used as the plot title. If None, the mnemonic name and description (if present) will be used as the title
- xrangetuple
Tuple of min, max datetime values to use as the plot range in the x direction.
- yrangetuple
Tuple of min, max datetime values to use as the plot range in the y direction.
- return_componentsbool
If True, return the plot as div and script components
- return_figbool
If True, return the plot as a bokeh Figure object
- plot_databool
If True, plot the data in the EdbMnemonic.data table
- plot_meanbool
If True, also plot the line showing the self.mean values
- plot_medianbool
If True, also plot the line showing the self.median values
- plot_maxbool
If True, also plot the line showing the self.max values
- plot_minbool
If True, also plot the line showing the self.min values
- Returns:
- objlist or bokeh.plotting.figure
If return_components is True, return a list containing [div, script] If return_figre is True, return the bokeh figure itself
- bokeh_plot_text_data(show_plot=False)[source]¶
Make basic bokeh plot showing value as a function of time.
- Parameters:
- show_plotboolean
A switch to show the plot in the browser or not.
- Returns:
- [div, script]list
List containing the div and js representations of figure.
- change_only_add_points()[source]¶
Tweak change-only data. Add an additional data point immediately prior to each original data point, with a value equal to that in the previous data point. This will help with filtering data based on conditions later, and will create a plot that looks more realistic, with only horizontal and vertical lines.
- daily_stats(sigma=3)[source]¶
Calculate the statistics for each day in the data contained in data[“data”]. Should we add a check for a case where the final block of time is <<1 day?
- Parameters:
- sigmaint
Number of sigma to use for sigma clipping
- full_stats(sigma=3)[source]¶
Calculate the mean/median/stdev of the full compliment of data
- Parameters:
- sigmaint
Number of sigma to use for sigma clipping
- interpolate(times)[source]¶
Interpolate data euvalues at specified datetimes.
- Parameters:
- timeslist
List of datetime objects describing the times to interpolate to
- plot_data_plus_devs(use_median=False, show_plot=False, savefig=False, out_dir='./', nominal_value=None, yellow_limits=None, red_limits=None, xrange=(None, None), yrange=(None, None), title=None, return_components=True, return_fig=False, plot_max=False, plot_min=False)[source]¶
Make basic bokeh plot showing value as a function of time. Optionally add a line indicating nominal (expected) value, as well as yellow and red background regions to denote values that may be unexpected. Also add a plot of the mean value over time and in a second figure, a plot of the devaition from the mean.
- Parameters:
- use_medianbool
If True, plot the median rather than the mean, as well as the deviation from the median rather than from the mean
- show_plotbool
If True, show plot on screen rather than returning div and script
- savefigbool
If True, file is saved to html file
- out_dirstr
Directory into which the html file is saved
- nominal_valuefloat
Expected or nominal value for the telemetry. If provided, a horizontal dashed line at this value will be added.
- yellow_limitslist
2-element list giving the lower and upper limits outside of which the telemetry value is considered non-nominal. If provided, the area of the plot between these two values will be given a green background, and that outside of these limits will have a yellow background.
- red_limitslist
2-element list giving the lower and upper limits outside of which the telemetry value is considered worse than in the yellow region. If provided, the area of the plot outside of these two values will have a red background.
- xrangetuple
Tuple of min, max datetime values to use as the plot range in the x direction.
- yrangetuple
Tuple of min, max datetime values to use as the plot range in the y direction.
- titlestr
Will be used as the plot title. If None, the mnemonic name and description (if present) will be used as the title
- return_componentsbool
If True, return the plot as div and script components
- return_figbool
If True, return the plot as a bokeh Figure object
- plot_maxbool
If True, also plot the line showing the self.max values
- plot_minbool
If True, also plot the line showing the self.min values
- Returns:
- objlist or bokeh.plotting.figure
If return_components is True, return a list containing [div, script] If return_figre is True, return the bokeh figure itself
- jwql.edb.engineering_database.add_limit_boxes(fig, yellow=None, red=None)[source]¶
Add green/yellow/red background colors
- Parameters:
- figbokeh.plotting.figure
Bokeh figure of the telemetry values
- yellowlist
2-element list of [low, high] values. If provided, the areas of the plot less than <low> and greater than <high> will be given a yellow background, to indicate an area of concern.
- redlist
2-element list of [low, high] values. If provided, the areas of the plot less than <low> and greater than <high> will be given a red background, to indicate values that may indicate an error. It is assumed that the low value of red is less than the low value of yellow, and that the high value of red is greater than the high value of yellow.
- Returns:
- figbokeh.plotting.figure
Modified figure with BoxAnnotations added
- jwql.edb.engineering_database.add_time_offset(offset, dt_obj)[source]¶
Add an offset to an input datetime object
- Parameters:
- offsetfloat
Number of seconds to be added
- dt_objdatetime.datetime
Datetime object to which the seconds are added
- Returns:
- objdatetime.datetime
Sum of the input datetime objects and the offset seconds.
- jwql.edb.engineering_database.calc_median_time(time_arr)[source]¶
Calcualte the median time of the input time_arr
- Parameters:
- time_arrnumpy.ndarray
1D array of datetime objects
- Returns:
- med_timedatetime.datetime
Median time, as a datetime object
- jwql.edb.engineering_database.change_only_bounding_points(date_list, value_list, starttime, endtime)[source]¶
For data containing change-only values, where bracketing data outside the requested time span may be present, create data points at the starting and ending times. This can be helpful with later interpolations.
- Parameters:
- date_listlist
List of datetime values
- value_listlist
List of corresponding mnemonic values
- starttimedatetime.datetime
Start time
- endtimedatetime.datetime
End time
- Returns:
- date_listlist
List of datetime values
- value_listlist
List of corresponding mnemonic values
- jwql.edb.engineering_database.change_only_stats(times, values, sigma=3)[source]¶
Calculate the mean/median/stdev as well as the median time for a collection of change-only data.
- Parameters:
- timeslist
List of datetime objects
- valueslist
List of values corresponding to times
- sigmafloat
Number of sigma to use for sigma-clipping
- Returns:
- meanvalfloat
Mean of values
- medianvalfloat
Median of values
- stdevvalfloat
Standard deviation of values
- jwql.edb.engineering_database.create_time_offset(dt_obj, epoch)[source]¶
Subtract input epoch from a datetime object and return the residual number of seconds
- Parameters:
- dt_objdatetime.datetime
Original datetiem object
- epochdatetime.datetime
Datetime to be subtracted from dt_obj
- Returns:
- objfloat
Number of seconds between dt_obj and epoch
- jwql.edb.engineering_database.get_mnemonic(mnemonic_identifier, start_time, end_time)[source]¶
Execute query and return an
EdbMnemonic
instance.The underlying MAST service returns data that include the datapoint preceding the requested start time and the datapoint that follows the requested end time.
- Parameters:
- mnemonic_identifierstr
Telemetry mnemonic identifiers, e.g.
SA_ZFGOUTFOV
- start_timeastropy.time.Time or datetime.datetime
Start time
- end_timeastropy.time.Time or datetime.datetime
End time
- Returns:
- mnemonicinstance of EdbMnemonic
EdbMnemonic object containing query results
- jwql.edb.engineering_database.get_mnemonic_info(mnemonic_identifier)[source]¶
Return the mnemonic description.
- Parameters:
- mnemonic_identifierstr
Telemetry mnemonic identifier, e.g.
SA_ZFGOUTFOV
- Returns:
- infodict
Object that contains the returned data
- jwql.edb.engineering_database.get_mnemonics(mnemonics, start_time, end_time)[source]¶
Query DMS EDB with a list of mnemonics and a time interval.
- Parameters:
- mnemonicslist or numpy.ndarray
Telemetry mnemonic identifiers, e.g.
['SA_ZFGOUTFOV', 'IMIR_HK_ICE_SEC_VOLT4']
- start_timeastropy.time.Time instance
Start time
- end_timeastropy.time.Time instance
End time
- Returns:
- mnemonic_dictdict
Dictionary. keys are the queried mnemonics, values are instances of EdbMnemonic
- jwql.edb.engineering_database.interpolate_datetimes(new_times, old_times, old_data)[source]¶
interpolate a set of datetime/value pairs onto another set of datetime objects
- Parameters:
- new_timesnumpy.ndarray
Array of datetime objects onto which the data will be interpolated
- old_timesnumpy.ndarray
Array of datetime objects associated with the input data values
- old_datanumpy.ndarray
Array of data values associated with
old_times
, which will be interpolated ontonew_times
- Returns:
- new_datanumpy.ndarray
Array of values interpolated onto
new_times
- jwql.edb.engineering_database.is_valid_mnemonic(mnemonic_identifier)[source]¶
Determine if the given string is a valid EDB mnemonic.
- Parameters:
- mnemonic_identifierstr
The mnemonic_identifier string to be examined.
- Returns:
- bool
Is mnemonic_identifier a valid EDB mnemonic?
- jwql.edb.engineering_database.mnemonic_inventory()[source]¶
Return all mnemonics in the DMS engineering database. No authentication is required, this information is public. Since this is a rather large and quasi-static table (~15000 rows), it is cached using functools.
- Returns:
- dataastropy.table.Table
Table representation of the mnemonic inventory.
- metadict
Additional information returned by the query.
- jwql.edb.engineering_database.process_mast_service_request_result(result, data_as_table=True)[source]¶
Parse the result of a MAST EDB query.
- Parameters:
- resultlist of requests.models.Response instances
The object returned by a call to
Mast.service_request_async
- data_as_tablebool
If
True
, return data as astropy table, else return as json
- Returns:
- dataastropy.table.Table
Table representation of the returned data.
- metadict
Additional information returned by the query