common_monitors

bad_pixel_monitor.py

This module contains code for the bad/dead pixel monitor.

The monitor calls the bad_pixel_mask.py module in the spacetelescope/jwst_reffiles package in order to identify bad pixels. (https://github.com/spacetelescope/jwst_reffiles)

The bad pixel mnemonics to be searched for are given in the jwst package: https://jwst-pipeline.readthedocs.io/en/stable/jwst/references_general/ references_general.html#data-quality-flags

The algorithms used to identify the various types of bad pixels were developed by the JWST Calibration Reference File Generation Tools working group, which was under the Calibration Working Group. This working group contained representatives from all instruments, and worked to identify common algorithms that all instruments could use to generate information to be placed in reference files. Details on the algorithms used here to identify bad pixels are detailed here: https://outerspace.stsci.edu/display/JWSTCC/Algorithm+details%3A+DQ+Init

The bad pixel search is composed of two different parts, each of which can be run independently.

1. Internal flat field exposures are used to search for DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels.

2. Dark current exposures are used to search for NOISY, HOT, RC, TELEGRAPH, and LOW_PEDESTAL pixels.

Both of these modules expect input data in at least 2 calibration states. In practice, the bad pixel monitor will search MAST for the appropriate dark current and flat field files. In both cases, a given file is considered useful if the uncal (and potentially the rate) versions of the file are present in the archive. For files where the uncal version only is found, the pipeline is run to produce the rate file.

Once a sufficient number of new flats or darks are identified, the bad pixel montior is called. The bad_pixel_file_thresholds.txt file contains a list of the minimum number of new files necessary to run the monitor for each instrument and aperture.

For the flat field files, the pipeline is run on the uncal files far enough to produce cosmic ray flagged (jump) files. These are also needed for the bad pixel search.

The jwst_reffiles bad_pixel_mask.py is run, and returns a map of bad pixels for each of the various bad pixel mnemonics. The bad_pixel_monitor then downloads the latest bad pixel mask in CRDS for the given instrument and detector/aperture, and this is compared to the new map of bad pixels. For each bad pixel mnemonic, any pixels flagged as bad that are not bad in the current reference file are saved to the (e.g. NIRCamBadPixelStats) database table.

Author

• Bryan Hilbert

Use

This module can be used from the command line as such:

python bad_pixel_monitor.py

Notes

The bad pixel flat that utilizes flat field ramps can’t be used with NIRCam since NIRCam has no internal lamp and therefore will not be taking any more internal flat field images. Could perhaps be used with a series of external undithered observations, but that’s something to think about later.

Templates to use: FGS_INTFLAT, NIS_LAMP, NRS_LAMP, MIR_DARK

class jwql.instrument_monitors.common_monitors.bad_pixel_monitor.BadPixels

Class for executing the bad pixel monitor.

This class will search for new (since the previous instance of the class) dark current and internal flat field files in the filesystem. It will loop over instrument/aperture combinations and find the number of new dark/flat files available. If there are enough, it will copy the files over to a working directory and run the monitor.

This will use the jwst_reffiles package to locate new bad pixels, which will be returned as a map. This map will be compared to the current bad pixel reference file (dq_init) in CRDS, and any the coordinates and type of any new bad pixels will be saved in a database table.

Raises:

ValueError

If NINT or DETECTOR is missing from input file header

ValueError

If an unrecognized bad pixel mnemonic is encountered

ValueError

If the number of uncal and rate files does not match

ValueError

If the most recent query search returns more than one entry

Attributes:

aperturestr

Aperture name of the data (e.g. NRCA1_FULL)

dark_query_startfloat

Date (in MJD) of the ending range of the previous MAST query where the bad pixel from darks monitor was run.

data_dirstr

Directory that contains the files copied from MAST to be used by the bad pixel monitor

detectorstr

Detector associated with the data (e.g. NRCA1)

flat_query_startfloat

Date (in MJD) of the ending range of the previous MAST query where the bad pixel from flats monitor was run.

instrumentstr

Name of the JWST instrument the data are from

nintsint

Number of integrations in the exposure

output_dirstr

Top level output directory associated with the bad pixel monitor, as retrieved from the JWQL config file

pixel_tablesqlalchemy table

Database table containing lists of bad pixels identified during runs of the bad pixel monitor

query_endfloat

MJD of the execution time of the bad pixel monitor. This is used as the ending time of the MAST query.

query_tablesqlalchemy table

Database table containing the history of MAST queries for the bad pixel monitor.

add_bad_pix(coordinates, pixel_type, files, obs_start_time, obs_mid_time, obs_end_time, baseline_file)

Add a set of bad pixels to the bad pixel database table

Parameters:

coordinatestuple

Tuple of two lists, containing x,y coordinates of bad pixels (Output of np.where call)

pixel_typestr

Type of bad pixel. e.g. dead, hot, and noisy

fileslist

List of fits files which were used to identify the bad pixels

obs_start_timedatetime.datetime

Observation time of the earliest file in files

obs_mid_timedatetime.datetime

Average of the observation times in files

obs_end_timedatetime.datetime

Observation time of the latest file in files

baseline_filestr

Name of baseline bad pixel file against which the new bad pixel population was compared

exclude_existing_badpix(badpix, pixel_type)

Given a set of coordinates of bad pixels, determine which of these pixels have been previously identified and remove them from the list

Parameters:

badpixtuple

Tuple of lists containing x and y pixel coordinates. (Output of numpy.where call)

pixel_typestr

Type of bad pixel being examined. Options are hot, dead, and noisy

Returns:

new_pixels_xlist

List of x coordinates of new bad pixels

new_pixels_ylist

List of y coordinates of new bad pixels

filter_query_results(results, datatype)

Filter MAST query results. For input flats, keep only those with the most common filter/pupil/grating combination. For both flats and darks, keep only those with the most common readout pattern.

Parameters:

resultslist

List of query results, as returned by mast_query()

datatypestr

Type of data being filtered. flat or dark.

Returns:

readpatt_filteredlist

Filtered list of query results.

get_metadata(filename)

Collect basic metadata from a fits file

Parameters:

filenamestr

Name of fits file to examine

get_possible_apertures()

Generate a list of possible SIAF apertures for the given instrument.

Returns:

possible_aperturelist

List of acceptible apertures for self.instrument

identify_tables()

Determine which database tables to use for a run of the bad pixel monitor

make_crds_parameter_dict()

Construct a paramter dictionary to be used for querying CRDS

Returns:

parametersdict

Dictionary of parameters, in the format expected by CRDS

map_uncal_and_rate_file_lists(uncal_files, rate_files, rate_files_to_copy, obs_type)

Copy uncal and rate files from the filesystem to the working directory. Any requested files that are not in the filesystem are noted and skipped. Return the file lists with skipped files removed.

Parameters:

uncal_fileslist

List of raw files to be copied

rate_fileslist

List of rate (slope) images to be copied. This list should correspond 1-to-1 with uncal_files. Any rate files that were not found in the MAST query should be set to None.

rate_files_to_copylist

Similar to rate_files but with the None entries omitted.

obs_typestr

Observation type (dark or flat). Used only for logging

Returns:

uncal_fileslist

List of the input raw files with any that failed to copy removed

rate_fileslist

List of the input rate files with any that failed to copy removed (if the uncal also failed) or set to None (if only the rate file failed)

most_recent_search(file_type='dark')

Query the query history database and return the information on the most recent query for the given aperture_name where the dark monitor was executed.

Parameters:

file_typestr

dark or flat. Specifies the type of file whose previous search time is queried.

Returns:

query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the dark monitor was run.

process(illuminated_raw_files, illuminated_slope_files, flat_file_count_threshold, dark_raw_files, dark_slope_files, dark_file_count_threshold)

The main method for processing darks. See module docstrings for further details.

Parameters:

illuminated_raw_fileslist

List of filenames (including full paths) of raw (uncal) flat field files. These should all be for the same detector and aperture.

illuminated_slope_fileslist

List of filenames (including full paths) of flat field slope files. These should all be for the same detector and aperture and correspond one-to-one with illuminated_raw_files. For cases where a raw file exists but no slope file, the slope file should be None

dark_raw_fileslist

List of filenames (including full paths) of raw (uncal) dark files. These should all be for the same detector and aperture.

dark_slope_fileslist

List of filenames (including full paths) of dark current slope files. These should all be for the same detector and aperture and correspond one-to-one with dark_raw_files. For cases where a raw file exists but no slope file, the slope file should be None

run(**kwargs)

Caller.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.bad_map_to_list(badpix_image, mnemonic)

Given a DQ image and a bad pixel mnemonic, create a list of (x,y) locations of this type of bad pixel in badpix_image

Parameters:

badpix_imagenumpy.ndarray

2D image of bad pixels (i.e. a DQ array)

mnemonicstr

The type of bad pixel to map. The mnemonic must be one of those in the JWST calibration pipeline’s list of possible mnemonics

Returns:

x_loclist

List of x locations within badpix_image containing mnemonic pixels.

y_loclist

List of x locations within badpix_image containing mnemonic pixels.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.check_for_sufficient_files(uncal_files, instrument_name, aperture_name, threshold_value, file_type)

From a list of files of a given type (flats or darks), check to see if there are enough files to call the bad pixel monitor. The number of files must be equal to or greater than the provided threshold_value.

Parameters:

uncal_fileslist

List of filenames

instrument_namestr

Name of JWST instrument (e.g. nircam) that the data are from. This is used only in logging statements

aperture_namestr

Name of aperture (e.g.`` NRCA1_FULL``) that the data are from. This is used only in logging statements

threshold_valueint

Minimum number of files required in order to run the bad pixel monitor

file_typestr

Either darks` or flats. This is used only in the logging statements.

Returns:

uncal_fileslist

List of sorted, unique files from the input file list. Set to None if the number of unique files is under the threshold

run_databool

Whether or not the bad pixel monitor will be called on these files.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.exclude_crds_mask_pix(bad_pix, existing_bad_pix)

Find differences between a set of newly-identified bad pixels and an existing set. Return a list of newly-discovered bad pixels that are not present in the existing set.

Parameters:

bad_pixnumpy.ndarray

2D array of bad pixel flags. Flags must correspond to the defintiions used by the JWST calibration pipeline

existing_bad_pixnumpy.ndarray

2D array of bad pixel flags. Flags must correspond to the definitions used by the JWST calibration pipeline

Returns:

new_bad_pixnumpy.ndarray

2D array of bad pixel flags contained in bad_pix but not existing_bad_pix

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.locate_rate_files(uncal_files)

Given a list of uncal (raw) files, generate a list of corresponding rate files. For each uncal file, if the rate file is present in the filesystem, add the name of the rate file (if a rateints file exists, use that) to the list of files. If no rate file is present, add None to the list.

Parameters:

uncal_fileslist

List of uncal files to use as the basis of the search

Returns:

rate_fileslist

List of rate files. This list corresponds 1-to-1 with uncal_files. Any missing rate files are listed as None.

rate_files_to_copylist

Same as rate_files but without the None entries. This is a list of only the rate files that exist in the filesystem

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.locate_uncal_files(query_result)

Given a MAST query result, locate the raw version (uncal.fits) of the listed files in the filesystem.

Parameters:

query_resultlist

MAST query results. List of dictionaries

Returns:

uncal_fileslist

List of raw file locations within the filesystem

bias_monitor.py

This module contains code for the bias monitor, which monitors the bias levels in dark exposures as well as the performance of the pipeline superbias subtraction over time.

For each instrument, the 0th group of full-frame dark exposures is saved to a fits file. The median signal levels in these images are recorded in the <Instrument>BiasStats database table for the odd/even rows/columns of each amp.

Next, these images are run through the jwst pipeline up through the reference pixel correction step. These calibrated images are saved to a fits file as well as a png file for visual inspection of the quality of the pipeline calibration. A histogram distribution of these images, as well as their collapsed row/column and sigma-clipped mean and standard deviation values, are recorded in the <Instrument>BiasStats database table.

Author

• Ben Sunnquist

• Maria A. Pena-Guerrero

Use

This module can be used from the command line as such:

python bias_monitor.py

class jwql.instrument_monitors.common_monitors.bias_monitor.Bias

Class for executing the bias monitor.

This class will search for new full-frame dark current files in the file system for each instrument and will run the monitor on these files. The monitor will extract the 0th group from the new dark files and output the contents into a new file located in a working directory. It will then perform statistical measurements on these files before and after pipeline calibration in order to monitor the bias levels over time as well as ensure the pipeline superbias is sufficiently calibrating new data. Results are all saved to database tables.

Attributes:

output_dirstr

Path into which outputs will be placed.

data_dirstr

Path into which new dark files will be copied to be worked on.

query_startfloat

MJD start date to use for querying MAST.

query_endfloat

MJD end date to use for querying MAST.

instrumentstr

Name of instrument used to collect the dark current data.

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL).

collapse_image(image)

Median-collapse the rows and columns of an image.

Parameters:

imagenumpy.ndarray

2D array on which to calculate statistics.

Returns:

collapsed_rowsnumpy.ndarray

1D array of the collapsed row values.

collapsed_columnsnumpy.ndarray

1D array of the collapsed column values.

determine_pipeline_steps()

Determines the necessary JWST pipelines steps to run on a given dark file.

Returns:

pipeline_stepscollections.OrderedDict

The pipeline steps to run.

extract_zeroth_group(filename)

Extracts the 0th group of a fits image and outputs it into a new fits file.

Parameters:

filenamestr

The fits file from which the 0th group will be extracted.

Returns:

output_filenamestr

The full path to the output file.

file_exists_in_database(filename)

Checks if an entry for filename exists in the bias stats database.

Parameters:

filenamestr

The full path to the uncal filename.

Returns:

file_existsbool

True if filename exists in the bias stats database.

get_amp_medians(image, amps)

Calculates the median in the input image for each amplifier and for odd and even rows/columns separately.

Parameters:

imagenumpy.ndarray

2D array on which to calculate statistics.

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)].

Returns:

amp_mediansdict

Median values for each amp. Keys are ramp numbers as strings with even/odd designation (e.g. '1_even').

identify_tables()

Determine which database tables to use for a run of the bias monitor.

image_to_png(image, outname)

Ouputs an image array into a png file.

Parameters:

imagenumpy.ndarray

2D image array.

outnamestr

The name given to the output png file.

Returns:

output_filenamestr

The full path to the output png file.

make_histogram(data)

Creates a histogram of the input data and returns the bin centers and the counts in each bin.

Parameters:

datanumpy.ndarray

The input data.

Returns:

countsnumpy.ndarray

The counts in each histogram bin.

bin_centersnumpy.ndarray

The histogram bin centers.

most_recent_search()

Query the query history database and return the information on the most recent query for the given aperture_name where the bias monitor was executed.

Returns:

query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the bias monitor was run.

process(file_list)

The main method for processing darks. See module docstrings for further details.

Parameters:

file_listlist

List of filenames (including full paths) to the dark current files.

run(**kwargs)

Caller.

dark_monitor.py

This module contains code for the dark current monitor, which performs some basic analysis to check whether the dark current behavior for the most recent set of input files is consistent with that from past files.

If enough new files for a given instrument/aperture combination (currently the files must be identified as dark current files in the exp_type header keyword) are present in the filesystem at the time the dark_monitor is called, the files are first run through the the appropriate pipeline steps to produce slope images.

A mean slope image as well as a standard deviation slope image is created by sigma-clipping on a pixel by pixel basis. The mean and standard deviation images are saved to a fits file, the name of which is entered into the <Instrument>DarkCurrent database table.

The mean slope image is then normalized by an existing baseline slope image, from the previous run of the monitor. New hot pixels are identified as those with normalized signal rates above a hot_threshold value. Similarly, pixels with normalized signal rates below a dead_threshold are flagged as new dead pixels.

The standard deviation slope image is also normalized by a baseline (historical) standard deviation image. Pixels with normalized values above a noise threshold are flagged as newly noisy pixels.

New hot, dead, and noisy pixels are saved to the DarkPixelStats database table.

Next, the dark current in the mean slope image is examined. A histogram of the slope values is created for the pixels in each amplifier, as well as for all pixels on the detector. In all cases, a Gaussian is fit to the histogram. Currently for NIRCam and NIRISS, a double Gaussian is also fit to the histogram from the entire detector.

The histogram itself as well as the best-fit Gaussian and double Gaussian parameters are saved to the DarkDarkCurrent database table.

Currently, there are three outputs from the dark monitor that are shown in the JWQL web app. First, the dark current histogram is plotted, along with a corresponding cumulative distribution function (CDF). The Gaussian fits are not currently shown.

Secondly, a trending plot of the mean dark current versus time is shown, where the mean value is the sigma-clipped mean across the detector in the mean slope image. Error bars on the plot show the sigma-clipped standard deviation across the detector.

Finally, the mean slope image is shown. Any new potential hot, dead, and noisy pixels that were identified are also shown on the mean slope image, in order to give an idea of where these pixels are located on the detector. To keep the image from becoming too busy, this is only done if the number of potential new bad pixels is under 1000. If more pixels that this are identified, that number is reported in the plot, but the pixels are not marked on the image.

Author

• Bryan Hilbert

Use

This module can be used from the command line as such:

python dark_monitor.py

class jwql.instrument_monitors.common_monitors.dark_monitor.Dark

Class for executing the dark current monitor.

This class will search for new (since the previous instance of the class) dark current files in the file system. It will loop over instrument/aperture combinations and find the number of new dark current files available. If there are enough, it will copy the files over to a working directory and run the monitor. This will create a mean dark current rate image, create a histogram of the dark current values, and fit several functions to the histogram. It will also compare the dark current image to a historical image in order to search for new hot or dead pixels. Results are all saved to database tables.

Parameters:

testingbool

For pytest. If True, an instance of Dark is created, but no other code is executed.

Raises:

ValueError

If encountering an unrecognized bad pixel type

ValueError

If the most recent query search returns more than one entry

Attributes:

output_dirstr

Path into which outputs will be placed

working_dirstr

Path into which new dark files will be copied to be worked on

query_startfloat

MJD start date to use for querying MAST

query_endfloat

MJD end date to use for querying MAST

instrumentstr

Name of instrument used to collect the dark current data

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL)

query_tablesqlalchemy table

Table containing the history of dark current queries to MAST for each instrument/aperture combination

pixel_tablesqlalchemy table

Table containing lists of hot/dead/noisy pixels found for each instrument/detector

stats_tablesqlalchemy table

Table containing dark current analysis results. Mean/stdev values, histogram information, Gaussian fitting results, etc.

add_bad_pix(coordinates, pixel_type, files, mean_filename, baseline_filename, observation_start_time, observation_mid_time, observation_end_time)

Add a set of bad pixels to the bad pixel database table

Parameters:

coordinatestuple

Tuple of two lists, containing x,y coordinates of bad pixels (Output of np.where call)

pixel_typestr

Type of bad pixel. Options are dead, hot, and noisy

fileslist

List of fits files which were used to identify the bad pixels

mean_filenamestr

Name of fits file containing the mean dark rate image used to find these bad pixels

baseline_filenamestr

Name of fits file containing the baseline dark rate image used to find these bad pixels

observation_start_timedatetime.datetime

Observation time of the earliest file in files

observation_mid_timedatetime.datetime

Average of the observation times in files

observation_end_timedatetime.datetime

Observation time of the latest file in files

create_mean_slope_figure(image, num_files, hotxy=None, deadxy=None, noisyxy=None, baseline_file=None, min_time='', max_time='')

Create and save a png containing the mean dark slope image, to be displayed in the web app

Parameters:

imagenumpy.ndarray

2D array of the dark slop image

num_filesint

Number of individual exposures that went into creating the mean slope image

hotxytup

2-tuple of lists that give x, y coordinates of possible new hot pixels

deadxytup

2-tuple of lists that give x, y coordinates of possible new hot pixels

noisyxytup

2-tuple of lists that give x, y coordinates of possible new hot pixels

baseline_filestr

Name of fits file containing the mean slope image to which image was compared when looking for new hot/dead/noisy pixels

min_timestr

Earliest observation time, in MJD, used in the creation of image.

max_timestr

Latest observation time, in MJD, used in the creation of image.

exclude_existing_badpix(badpix, pixel_type)

Given a set of coordinates of bad pixels, determine which of these pixels have been previously identified and remove them from the list

Parameters:

badpixtuple

Tuple of lists containing x and y pixel coordinates. (Output of numpy.where call)

pixel_typestr

Type of bad pixel being examined. Options are hot, dead, and noisy

Returns:

new_pixels_xlist

List of x coordinates of new bad pixels

new_pixels_ylist

List of y coordinates of new bad pixels

exclude_too_few_groups(result_list)

Given a list of mast query results, go through and exlclude files that have too few groups to be useful

Parameters:

result_listlist

List of dictionaries containing a MAST query result

Returns:

filtered_resultslist

List of dictionaries with files containing too few groups excluded

find_hot_dead_pixels(mean_image, comparison_image, hot_threshold=2.0, dead_threshold=0.1)

Create the ratio of the slope image to a baseline slope image. Pixels in the ratio image with values above hot_threshold will be marked as hot, and those with ratio values less than dead_threshold will be marked as dead.

Parameters:

mean_imagenumpy.ndarray

2D array containing the slope image from the new data

comparison_imagenumpy.ndarray

2D array containing the baseline slope image to compare against the new slope image.

hot_thresholdfloat

(mean_image / comparison_image) ratio value above which a pixel is considered hot.

dead_thresholdfloat

(mean_image / comparison_image) ratio value below which a pixel is considered dead.

Returns:

hotpixtuple

Tuple (of lists) containing x,y coordinates of newly hot pixels

deadpixtuple

Tuple (of lists) containing x,y coordinates of newly dead pixels

get_baseline_filename()

Query the database and return the filename of the baseline (comparison) mean dark slope image to use when searching for new hot/dead/noisy pixels. For this we assume that the most recent baseline file for the given detector is the one to use.

Returns:

filenamestr

Name of fits file containing the baseline image

get_metadata(filename)

Collect basic metadata from a fits file

Parameters:

filenamestr

Name of fits file to examine

identify_tables()

Determine which database tables to use for a run of the dark monitor

most_recent_search()

Query the query history database and return the information on the most recent query for the given aperture_name where the dark monitor was executed.

Returns:

query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the dark monitor was run.

noise_check(new_noise_image, baseline_noise_image, threshold=1.5)

Create the ratio of the stdev (noise) image to a baseline noise image. Pixels in the ratio image with values above threshold will be marked as newly noisy.

Parameters:

new_noise_imagenumpy.ndarray

2D array containing the noise image from the new data

baseline_noise_imagenumpy.ndarray

2D array containing the baseline noise image to compare against the new noise image.

thresholdfloat

(new_noise_image / baseline_noise_image) ratio value above which a pixel is considered newly noisey.

Returns:

noisytuple

Tuple (of lists) of x,y coordinates of newly noisy pixels

overplot_bad_pix(pix_type, coords, values)

Add a scatter plot of potential new bad pixels to the plot

Parameters:

pix_typestr

Type of bad pixel. “hot”, “dead”, or “noisy”

coordstup

2-tuple of lists, containing the x and y coordinates of the bad pixels

valueslist

Values in the mean dark image at the locations of the bad pixels

Returns:

legend_itemtup

Tuple of legend text and associated plot. Will be converted into a LegendItem and added to the plot legend

process(file_list)

The main method for processing darks. See module docstrings for further details.

Parameters:

file_listlist

List of filenames (including full paths) to the dark current files

read_baseline_slope_image(filename)

Read in a baseline mean slope image and associated standard deviation image from the given fits file

Parameters:

filenamestr

Name of fits file to be read in

Returns:

mean_imagenumpy.ndarray

2D mean slope image

stdev_imagenumpy.ndarray

2D stdev image

run(**kwargs)

Caller.

save_mean_slope_image(slope_img, stdev_img, files, min_time, max_time)

Save the mean slope image and associated stdev image to a file

Parameters:

slope_imgnumpy.ndarray

2D array containing the mean slope image

stdev_imgnumpy.ndarray

2D array containing the stdev image associated with the mean slope image.

fileslist

List of input files used to construct the mean slope image

min_timestr

Earliest observation time, in MJD, corresponding to files.

max_timestr

Latest observation time, in MJD, corresponding to files.

Returns:

output_filenamestr

Name of fits file to save mean and stdev images within

shift_to_full_frame(coords)

Shift the input list of pixels from the subarray coordinate system to the full frame coordinate system

Parameters:

coordstup

(x, y) pixel coordinates in subarray coordinate system

Returns:

coordstup

(x, y) pixel coordinates in full frame coordinate system

split_files_into_sub_lists(files, start_times, end_times, integration_list, threshold)

Given a list of filenames and a list of the number of integrations within each, split the files into sub-lists, where the files in each list have a total number of integrations that is just over the given threshold value.

General assumption: Keeping files in different epochs separate is probably more important than rigidly enforcing that the required number of integrations is reached.

When dividing up the input files into separate lists, we first divide up by epoch, where the start/end of epochs are defined as times where DARK_MONITOR_BETWEEN_EPOCH_THRESHOLD_TIME days pass without any new data appearing. Each epoch is then potentially subdivided further based on the threshold number of integrations (not exposures). The splitting does not operate within files. For example, if the threshold is 2 integrations, and a particular file contains 5 integrations, then the dark monitor will be called once on that file, working on all 5 integrations.

At the end of the epoch, if the final group of file(s) do not have enough integrations to reach the threshold, they are ignored since there is no way to know if there are more files in the same epoch that have not yet been taken. So the files are ignored, and the query end time will be adjusted such that these files will be found in the next run of the monitor.

Dark calibration plans per instrument: NIRCam - for full frame, takes only 2 integrations (150 groups) once per ~30-50 days.

for subarrays, takes 5-10 integrations once per 30-50 days

team response -

NIRISS - full frame - 2 exps of 5 ints within each 2 week period. No requirement for

the 2 exps to be taken at the same time though. Could be separated by almost 2 weeks, and be closer to the darks from the previous or following 2 week period.

subarrays - 30 ints in each month-long span

MIRI - 2 ints every 2 hours-5 days for a while, then 2 ints every 14-21 days

team response - monitor should run on each exp separately. It should also throw out

the first integration of each exp.

NIRSpec - full frame 5-6 integrations spread over each month

subarray - 12 ints spread over each 2 month period

FGS - N/A

Parameters:

fileslist

List of filenames

integration_listlist

List of integers describing how many integrations are in each file

start_timeslist

List of MJD dates corresponding to the exposure start time of each file in files

end_timeslist

List of MJD dates corresponding to the exposures end time of each file in files

integration_listlist

List of the number of integrations for each file in files

thresholdint

Threshold number of integrations needed to trigger a run of the dark monitor

stats_by_amp(image, amps)

Calculate statistics in the input image for each amplifier as well as the full image

Parameters:

imagenumpy.ndarray

2D array on which to calculate statistics

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)]

Returns:

amp_meansdict

Sigma-clipped mean value for each amp. Keys are amp numbers as strings (e.g. '1')

amp_stdevsdict

Sigma-clipped standard deviation for each amp. Keys are amp numbers as strings (e.g. '1')

gaussian_paramsdict

Best-fit Gaussian parameters to the dark current histogram. Keys are amp numbers as strings. Values are three-element lists [amplitude, peak, width]. Each element in the list is a tuple of the best-fit value and the associated uncertainty.

gaussian_chi_squareddict

Reduced chi-squared for the best-fit parameters. Keys are amp numbers as strings

double_gaussian_paramsdict

Best-fit double Gaussian parameters to the dark current histogram. Keys are amp numbers as strings. Values are six- element lists. (3-elements * 2 Gaussians). [amplitude1, peak1, stdev1, amplitude2, peak2, stdev2] Each element of the list is a tuple containing the best-fit value and associated uncertainty.

double_gaussian_chi_squareddict

Reduced chi-squared for the best-fit parameters. Keys are amp numbers as strings

histdict

Dictionary of 1D arrays of histogram values

binsdict

Dictionary of 1D arrays of bin centers that match the hist values.

edb_telemetry_monitor.py

Engineering Database Mnemonics Trending Monitor (EDB Trending Monitor)

This module contains code for the Engineering Database Telemetry monitor. For a given mnemonic, this monitor retrieves telemetry data from the EDB, filters the data based on optional conditions, calculates some basic statistics on the data, and plots the data.

There is a list of mnemonics to monitor for each instrument, in the form of a json file. For each mnemonic, the file specifies the conditions for filtering the data. This may include filtering based on the values of the mnemonic’s own data (e.g. keep only entries where the voltage is < 0.25V), and/or filtering based on the values of dependency mnemonics. For example, keep data for mnemonic A only when mnemonic B is > 0.25V and mnemonic C is less than 38K.

Statistics

After filtering the data, the monitor calcualtes statistics. The monitor supports several different types averaging. These include:

1. daily_means - This is designed for mnemonics whose values do not change much over the course of a day. In this case, mnemonic data is retrieved over a small amount of time each day (e.g. 12:00 - 12:15). From these data, a daily mean is calculated. For all other types of telemetry, the EDB queries span the full day.

2. block_means - These are mnemonics where the user wishes to see mean values associated with each block of entries in the retrieved and filtered data. For example, you want to examine a voltage at times when some other current is less than 0.25A. The script will read in all telemetry data, and filter out data points for times where the current did not meet the criteria. It will then calculate the mean of each remaining block of continuous good data. So if the data were good from 2:00 to 2:30, then bad until 3:00, and good again from 3:00-4:00, then the monitor will calculate a mean value for the 2:00-2:30 period, and a mean from the 3:00-4:00 period.

3. time_interval - Mnemonics in this category have their data retrieved and filtered, and then averaged over the requested time interval. For example, if the user sets a time interval of 5 minutes, then the monitor caculates the mean value within each 5-minute block of the total time range of the data, and plots the average values.

4. every_change - This is the most complex case. Mnemonics in this category have their data filtered and organized based on the value of a secondary mnemonic. For example, the IMIR_HK_GW14_POS_RATIO returns a measure of the position of MIRI’s grating wheel. We can plot this position as a function of the commanded location of the grating wheel, which is provided by IMIR_HK_GW14_CUR_POS. In this case, the monitor will loop over the commanded positions and for each, gather the measured position information. The measured positions associated with each commanded position will then be plotted separately. Note that this use of “every change” is separate from the idea of every-change telemetry, in which telemetry points are only generated at times when the telemetry value changes. Some of the mnemonics in the EDB do contain change-only telemetry data, but this should be largely invisible to the EDB Telemetry Monitor user.

5. all - In this case, no averaging is done. (Although filtering is still done) All filtered data are kept as they are retrived from the EDB, and plotted without any modification.

6. all+daily_means - This is a combination of the “all” and “daily_means” cases above. All data points are retrieved from the EDB and optionally filtered by dependencies. Then daily means are calculated. Both the full set of data and the daily means are plotted, along with deviations from the mean.

7. all+block_means - This is a combination of the “all” and “block_means” cases above. All data points are retrieved from the EDB and optionally filtered by dependencies. Then means for each block of good data are calculated. Both the full set of data and the means are plotted, along with deviations from the mean.

8. all+time_interval - This is a combination of the “all” and “time_interval” cases above. All data points are retrieved from the EDB and optionally filtered by dependencies. Then means are calculated for each block of time lasting the duration of the time interval. Both the full set of data and the means are plotted, along with deviations from the mean.

JSON file format

The type of data averaging is at the top level of the JSON file. Values must match the 8 types described above. The entry for each mnemonic has several pieces of information, described below.

• name: Name of the mnemonic as it appears in the EDB.

• database_id Optional name to use in the plot title for this mnemonic. Any averaged data saved to the JWQL database will be saved under this name if it is present.

• description: Summary describing the data contained in the mnemonic. Placed in the plot title.

• dependency: This is a list of mnemonics and conditions that will be used to filter the data

• plot_data: Description of how the data are to be plotted. There are two options: “nominal”, in which case the mnemonic data are plotted as-is, and “*<mnem>” where <mnem> is the name of another mnemonic. In this case, the data for this second mnemonic are retrieved using the same dependencies as the primary mnemonic. The primary mnemonic and this second mnemonic are then multiplied together and plotted. This option was designed around plotting power as the product of current and voltage.

A further option for the “plot_data” field is the addition of a comma-separated list of statistics to be overplotted. Options are: “mean”, “median”, “max”, and “min”. Note that this is a little confusing, because in many cases the menmonic’s data will already contain the median value of the data (and the original data as returned from the EDB will not be available). The monitor realized this though, so if you specify “mean” for a mnemonic in the “daily_mean” list, it will simply plot the same data twice, on top of itself.

As an example, in order to plot the daily mean and maximum values of the product of SE_ZIMIRICEA and SE_ZBUSVLT, the plot_data entry would be: “*SE_ZBUSVLT,max”. If you also wanted to plot the minimum daily value, the entry would be: “*SE_ZBUSVLT,max,min”. And similarly, to plot SE_ZIMIRICEA on its own (not as a product), the plot_data entries shown above would become: “nominal,max” and “nominal,max,min”.

• nominal_value: Optional. The “expected” value for this mnemonic. If provided, a horizontal dashed line will be added at this value.

• yellow_limits: Optional. This is a list of two values that describe the lower and upper limits of the expected range of the mnemonic’s value. If these are present, a green background is added to the plot at values between these limits. Outside of these limits, the background is yellow.

• red_limits: Optional. Similar to yellow_limits above. In this case the lower and upper limits represent the thresholds outside of which there may be a problem. In this case, the background of the plot outside of these values is red.

• plot_category: This is the name of the tab on the website into which this plot should be placed.

• mean_time_block: Optional. This is only used for time_interval mnemonics. It describes the length of time over which to bin and average the data. The value consists of a number and a unit of time: e.g. “15_minutes”

Below we present details on how to construct json entries for these specific cases.

“daily_means” entries

Here is an example of two daily_mean telemetry entries in the json file. In both, SE_ZIMIRICEA values are retrieved. For the first plot, data are only kept for the times where the following dependencies are true:

1. SE_ZIMIRICEA is > 0.2 A

2. IMIR_HK_ICE_SEC_VOLT1 is < 1 V

3. IMIR_HK_IMG_CAL_LOOP is OFF

4. IMIR_HK_IFU_CAL_LOOP is OFF

5. IMIR_HK_POM_LOOP is OFF

For the second plot, data are kept for the times where:

1. IMIR_HK_ICE_SEC_VOLT1 is > 25 V

Note that the “database_id” entry is used to differentiate the plot labels of these two cases, as well as their entries in the JWQLDB. In both cases, the data are plotted as the product of SE_ZIMIRICEA and SE_ZBUSVLT. In the second case, the maximum daily value is also plotted. Both plots are placed in the power tab on the webpage.

{
    "daily_means": [
        {
            "name": "SE_ZIMIRICEA",
            "database_id": "SE_ZIMIRICEA_NO_OPS",
            "description": "ICE drive current (no ops)",
            "dependency": [
                {
                    "name": "SE_ZIMIRICEA",
                    "relation": ">",
                    "threshold": 0.2
                },
                {
                    "name": "IMIR_HK_ICE_SEC_VOLT1",
                    "relation": "<",
                    "threshold": 1
                },
                {
                    "name": "IMIR_HK_IMG_CAL_LOOP",
                    "relation": "=",
                    "threshold": "OFF"
                },
                {
                    "name": "IMIR_HK_IFU_CAL_LOOP",
                    "relation": "=",
                    "threshold": "OFF"
                },
                {
                    "name": "IMIR_HK_POM_LOOP",
                    "relation": "=",
                    "threshold": "OFF"
                }
            ],
            "plot_data": "*SE_ZBUSVLT",
            "nominal_value": 7.57,
            "yellow_limits": [7.0, 8.13],
            "plot_category": "power"
        },
        {
            "name": "SE_ZIMIRICEA",
            "database_id": "SE_ZIMIRICEA_OPS",
            "description": "ICE drive current (ops)",
            "dependency": [
                {
                    "name": "IMIR_HK_ICE_SEC_VOLT1",
                    "relation": ">",
                    "threshold": 25
                }
            ],
            "plot_data": "*SE_ZBUSVLT,max",
            "nominal_value": 11.13,
            "yellow_limits": [10.23, 12.02],
            "plot_category": "power"
        }
    ]
}

For a case with no dependencies, the “dependencies” keyword can be left empty:

{
    "name": "INRC_ICE_DC_VOL_P5_DIG",
    "description": "ICE HK +5V voltage for digital electronics",
    "dependency": [],
    "plot_data": "nominal",
    "yellow_limits": [4.99, 5.04],
    "red_limits": [4.5, 5.5],
    "plot_category": "ice_voltage"
}

“block_means” entries

In the example shown below, we want to plot IMIR_HK_ICE_SEC_VOLT1 at times when it has values higher than 25V. In this case, the EDB monitor will find times when the voltage is under the 25V limit. These times will separate the blocks of time when the voltage does meet the threshold value. It then calculates and plots the median voltage within each of these blocks.

"block_means":[
    {
        "name": "IMIR_HK_ICE_SEC_VOLT1",
        "database_id": "IMIR_HK_ICE_SEC_VOLT1_OPS",
        "description": "ICE Secondary Voltage (HV) 1",
        "dependency": [
            {
                "name": "IMIR_HK_ICE_SEC_VOLT1",
                "relation": ">",
                "threshold": 25
            }
        ],
        "plot_data": "nominal",
        "nominal_value": 39.24,
        "yellow_limits": [39.14, 39.34],
        "plot_category": "ICE_voltage"
    }

“time_interval” entries

For mnemonics to be averaged over some time period, use the “mean_time_block” entry. The value of mean_time_block should be a number followed by an underscore and a unit of time. Currently, the unit must contain one of “min”, “sec”, “hour”, or “day”. The monitor looks for one of these strings within the mean_time_block entry, meaning that “second”, “seconds”, “minutes”, “minute”, “hours”, “days”, etc are all valid entries.

In the example below, the EDB monitor will bin the SE_ZINRCICE1 data into 5 minute blocks, and calculate and plot the mean of each block.

"time_interval": [
    {
        "name": "SE_ZINRCICE1",
        "description": "ICE1 current",
        "mean_time_block": "5_min",
        "dependency": [],
        "plot_data": "nominal",
        "yellow_limits": [0.36, 0.8],
        "red_limits": [0, 1.367],
        "plot_category": "box_current"
    }
]

“every_change” entries

This is a complex case and at the moment is customized for the MIRI filter wheel position mnemonics such as IMIR_HK_FW_POS_RATIO. In this case, the EDB monitor will retrieve data for the filter wheel position, which is a float at each time. It will also retrive the commmanded position of the filter wheel, which is a string at each time (e.g. OPEN, CLOSED). It then divides the filter wheel postiion data into groups based on the value of the commanded position (i.e. group together all of the postion data when the commanded position is OPEN). It then computes the median value of the filter position within each continuous block of time where the commanded position is constant. This median value is then normalized by the expected location value (retrieved from constants.py). One line is plotted for each commanded position.

"every_change": [
    {
        "name": "IMIR_HK_FW_POS_RATIO",
        "description": "FW normalized position sensor voltage ratio",
        "dependency": [
            {
                "name": "IMIR_HK_FW_CUR_POS",
                "relation": "none",
                "threshold": 0
            }
        ],
        "plot_data": "nominal",
        "yellow_limits": [-1.6, 1.6],
        "plot_category": "Position_sensors"
    }
]

“all” entries

In this case, no grouping or averaging of data from the EDB are done. Data are retrieved from the EDB, filtered by any dependencies, and plotted.

"all": [
    {
        "name": "SE_ZINRCICE1",
        "description": "ICE1 current",
        "dependency": [],
        "plot_data": "nominal",
        "yellow_limits": [0.36, 0.8],
        "red_limits": [0, 1.367],
        "plot_category": "box_current"
    },
    {
        "name": "SE_ZINRCICE2",
        "description": "ICE2 current",
        "dependency": [],
        "plot_data": "nominal",
        "yellow_limits": [0.38, 0.86],
        "red_limits": [0, 1.372],
        "plot_category": "box_current"
    }
]

“all+daily_means” entries

This is a combination of the “daily_means” and “all” cases above.

“all+block_means” entries

This is a combination of the “all” and “block_means” cases above.

“all+time_interval” entries

This is a combination of the “all” and “time_interval” cases above.

Summary of the EDB monitor operation

The monitor is set up to find the total span of time over which the plots are requested (with the default being contolled by the value in jwql.utils.constants). It loops over each mnemonic and breaks the total time to be queried up into 24-hour long blocks. It then queries the EDB once for each day-long block, filters the data based on any dependencies given and then calculates statistics. Breaking the queries up into day-long blocks is done in order to avoid having the EDB return a very large table, which could cause memory problems, or slow the monitor down. This is a possibility because in some cases, mnemonic values are present at cadences of > 1 Hz.

After data are filtered and averaged (and combined with the data of a second mnemonic if they are being plotted as a product), any new data are saved to the JWQL database. This will prevent having to repeat the calculations during future queries. For mnemonics where no averaging is done, we do not save anything in the JWQL databases, in order to save memory.

Each time a query is initiated, the JWQL database is checked first and any relevent data are retrieved. In this way, we only query the EDB for new data.

The monitor creates one plot for each specified mnemonic. These plots are organized into “plot_categories”, as seen in the json examples above. All plots for a given category are placed together in a bokeh gridplot. Each gridplot (i.e. plot_category) is then placed in a separate bokeh tab, in order to try and keep related plots together while not overwhelming the user with too many plots at once. The tabbed plots are written out to a json file. When the user clicks on the EDB Telemetry Monitor link on the web app, this json file is read in and embedded into the html file that is displayed. With this method, EDB queries and data calculations are all done asynchronously, which means that the EDB Telemetry Monitor web page shoudl be fast to load.

Author

• Bryan Hilbert

Use

This module can be called from the command line like this:

python edb_telemetry_monitor.py

class jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.EdbMnemonicMonitor

Class for executing the EDB Telemetry Monitor

This class will search for and retrieve new telemetry data associated with the given mnemonics from the engineering database. These data will be filtered based on dependency menmonic details, and optioanlly averaged over some time period. These data will then be combined with data from previous runs of the EDB Telemetry Monitor, which have been stored in the JWQL database. The resulting data is then plotted.

Raises:

NotImplementedError

If multiple dependencies are provided for an “every_change” mnemonic, or if the dependency values are not strings

ValueError

If the user requests to plot every_change data as the product of two mnemonics

ValueError

If the user gives a list of mnemonics to query (rather than getting them from a json file), but starting and ending dates for the plot are not specified

NotImplementedError

If the user specifies a plot type other than “nominal” or the product of two mnemonics

ValueError

If the user calls plot_every_change_data and requests multiple output types for the resulting figure

Attributes:

query_resultsdict

Dictionary containing EDB query results for all mnemonics for the current session.

figuresdict

Dictionary of Bokeh figures. Keys are the plot_category, and values are lists of Bokeh figure objects.

history_tablesqlalchemy table

Table containing a history of the queries made for the mnemonic type

_usenamestr

Key to use when specifying the mnemonic’s identity

query_cadencedatetime.datetime

How often the EDB should be queried. Currently set to 1 day.

plot_output_dirstr

Directory into which the json file containing the gridded plots should be saved

instrumentstr

Name of the instrument whose mnemonics are being investigated

_plot_startdatetime.datetime

Fallback date for the plot start. Only used if the plot contains no data.

_plot_enddatetime.datetime

Fallback date for the plot end. Only used if the plot contains no data.

requested_start_timedatetime.datetime

Earliest start time for the current run of the EDB Monitor

requested_end_timedatetime.datetime

Latest end time for the current run of the EDB Monitor

add_figure(fig, key)

Add Bokeh figure to the dictionary of figures

Parameters:

figbokeh.plotting.figure

Plot of a single mnemonic

keystr

Key under which to store the plot

add_new_block_db_entry(mnem, query_time)

Add a new entry to the database table for any kind of telemetry type other than “all” (which does not save data in the database) and “every_change” (which needs a custom table.)

Parameters:

mnemjwql.edb.engineering_database.EdbMnemonic

Mnemonic information

query_timedatetime.datetime

Start time of the query

add_new_every_change_db_entry(mnem, mnem_dict, dependency_name, query_time)

Add new entries to the database table for “every change” mnemonics. Add a separate entry for each dependency value.

Parameters:

mnemstr

Name of the mnemonic whose data is being saved

mnem_dictdict

Dictionary containing every_change data as output by organize_every_change()

dependency_namestr

Name of mnemonic whose values the changes in mnemonic are based on

query_timedatetime.datetime

Start time of the query

calc_timed_stats(mnem_data, bintime, sigma=3)

Not currently used. Calculate stats for telemetry using time-based averaging. This works on data that have potentially been filtered. How do we treated any breaks in the data due to the filtering? Enforce a new bin at each filtered block of data? Blindly average by time and ignore any missing data due to filtering? The former makes more sense to me

Parameters:

mnem_datajwql.edb.engineering_database.EdbMnemonic

Mnemonic data to be averaged

bintimeastropy.time.Quantity

Time to use for binning and averaging data

Returns:

all_meanslist

List of mean values

all_medslist

List of median values

all_stdevslist

List of stadnard deviations

all_timeslist

List of times associated with the means, medians, and standard deviations

execute(**kwargs)

Caller.

filter_telemetry(mnem, data, dep_list)

Filter telemetry data for a single mnemonic based on a list of conditions/dependencies, as well as a time.

Parameters:

mnemstr

Name of the mnemonic whose dependencies will be queried

datajwql.edb.engineering_database.EdbMnemonic

Information and query results for a single mnemonic

dep_listlist

List of dependencies for a given mnemonic. Each element of the list is a dictionary containing the keys: name, relation, and threshold. In nominal operations, these are read out of the json file listing the mnemonics to be monitored.

Returns:

filteredjwql.edb.engineering_database.EdbMnemonic

Filtered information and query results for a single mnemonic

find_all_changes(mnem_data, dep_list)

Identify indexes of data to create separate blocks for each value of the condition. This is for the “every_change” mnemonics, where we want to create a mean value for all telemetry data acquired for each value of some dependency mnemonic.

For now, this function assumes that we only have one dependency. I’m not sure how it would work with multiple dependencies.

Parameters:

mnem_datajwql.edb.engineering_database.EdbMnemonic

EDBMnemonic instance to be searched

dep_listlist

List of dependency mnemonic names. Currently should be a 1-element list

Returns:

mnem_datajwql.edb.engineering_database.EdbMnemonic

EDBMnemonic instance that was input, with `blocks` and `every_change_values` attributes added.

generate_query_start_times(starting_time)

Generate a list of starting and ending query times such that the entire time range is covered, but we are only querying the EDB for one day’s worth of data at a time. Start times are once per day between the previous query time and the present. End times are the start times plus the query duration.

Parameters:

starting_timedatetime.datetime

Datetime specifying the earliest time to query the EDB

Returns:

query_starting_timeslist

List of datetime objects giving start times for EDB queries on a daily basis

query_ending_timeslist

List of datetime objects giving ending times for EDB queries on a daily basis

get_dependency_data(dependency, starttime, endtime)

Find EDB data for the mnemonic listed as a dependency. Keep a dcitionary up to date with query results for all dependencies, in order to minimize the number of queries we have to make. Return the requested dependency’s time and data values in a dictionary.

Parameters:

dependencydict

The name of the mnemonic to seach for should be the value associated with the “name” key.

starttimedatetime.datetime

Staritng time for the query

endtimedatetime.datetime

Ending time for the query

Returns:

dep_mnemonicdict

Data for the dependency mnemonic. Keys are “dates” and “euvalues”. This is essentially the data in the `data` attribute of an EDBMnemonic instance

get_history(mnemonic, start_date, end_date, info={}, meta={})

Retrieve data for a single mnemonic over the given time range from the JWQL database (not the EDB).

Parameters:

mnemonicstr

Name of mnemonic whose data is to be retrieved

start_datedatetime.datetime

Beginning date of data retrieval

end_datedatetime.datetime

Ending date of data retrieval

infodict

Info dictionary for an EDBMnemonic instance.

metadict

Meta dictionary for an EDBMnemonic instance.

Returns:

histjwql.edb.engineering_database.EdbMnemonic

Retrieved data

get_history_every_change(mnemonic, start_date, end_date)

Retrieve data from the JWQL database for a single mnemonic over the given time range for every_change data (e.g. IMIR_HK_FW_POS_RATIO, where we need to calculate and store an average value for each block of time where IMIR_HK_FW_CUR_POS has a different value. This has nothing to do with ‘change-only’ data as stored in the EDB.

Parameters:

mnemonicstr

Name of mnemonic whose data is to be retrieved

start_datedatetime.datetime

Beginning date of data retrieval

end_datedatetime.datetime

Ending date of data retrieval

Returns:

histdict

Retrieved data. Keys are the value of the dependency mnemonic, and each value is a 3-tuple. The tuple contains the times, values, and mean value of the primary mnemonic corresponding to the times that they dependency mnemonic has the value of the key.

get_mnemonic_info(mnemonic, starting_time, ending_time, telemetry_type)

Wrapper around the code to query the EDB, filter the result, and calculate appropriate statistics for a single mnemonic

Parameters:

mnemonicdict

Dictionary of information about the mnemonic to be processed. Dictionary as read in from the json file of mnemonics to be monitored.

starting_timedatetime.datetime

Beginning time for query

ending_timedatetime.datetime

Ending time for query

telemetry_typestr

How the telemetry will be processed. This is the top-level heirarchy from the json file containing the mnemonics. e.g. “daily_means”, “every_change”

Returns:

good_mnemonic_datajwql.edb.engineering_database.EdbMnemonic

EdbMnemonic instance containing filtered data for the given mnemonic

identify_tables(inst, tel_type)

Determine which database tables to use for a given type of telemetry.

Parameters:

inststr

Name of instrument (e.g. nircam)

tel_typestr

Type of telemetry. This comes from the json file listing all mnemonics to be monitored. Examples include “every_change”, “daily”, “all”, etc

most_recent_search(telem_name)

Query the database and return the information on the most recent query, indicating the last time the EDB Mnemonic monitor was executed.

Parameters:

telem_namestr

Mnemonic to search for

Returns:

query_resultdatetime.datetime

Date of the ending range of the previous query

multiday_mnemonic_query(mnemonic_dict, starting_time_list, ending_time_list, telemetry_type)

Wrapper function. In order to avoid any issues with giant tables, for a given mnemonic we query the EDB for only one day’s worth of data at a time. For each day we retrieve the data, retrieve the data for any dependencies, filter the data based on the dependencies, and then perform any requested averaging before moving on to the next day.

Parameters:

mnemonic_dictdict

Dictionary of information for a single mnemonic. This comes from the json file describing all mnemonics to be monitored

starting_time_listlist

List of datetime values indicating beginning query times

ending_time_listlist

List of datetime values indicating the end time of each query

telemetry_typestr

Type of telemetry being retrieved. This is from the top-level of the json file describing all mnemonics to be monitored. Examples include “every_change”, “daily”, “all”.

Returns:

all_datajwql.edb.engineering_database.EdbMnemonic

EDBMnemonic instance containing the mnemonic’s filtered, averaged data spanning the entire time range between the earliest start time and latest end time. Note that if averaging is done, then the data table in this instance contains the averaged data. The original data is not returned.

run(instrument, mnemonic_dict, plot_start=None, plot_end=None)

Run the monitor on a single mnemonic.

Parameters:

instrumentstr

Instrument name (e.g. nircam)

mnemonic_dictdict

Dictionary of information for a single mnemonic. Keys include: “name”, “description”, “depenency”, “plot_data”, “yellow_limits”, “red_limits”, “plot_category”. In normal operation, this is read in from the json file that lists all mnemonics to be monitored

plot_startdatetime.datetime

Starting time for the output plot

plot_enddatetime.datetime

Ending time for the output plot

tabbed_figure(ncols=2)

Create a tabbed object containing a panel of gridded plots in each tab.

Parameters:

ncolsint

Number of columns of plots in each plot tab

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.add_every_change_history(dict1, dict2)

Combine two dictionaries that contain every change data. For keys that are present in both dictionaries, remove any duplicate entries based on date.

Parameters:

dict1dict

First dictionary to combine

dict2dict

Second dictionary to combine

Returns:

combinedcollections.defaultdict

Combined dictionary

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.calculate_statistics(mnemonic_instance, telemetry_type)

Wrapper function around the various methods that can be used to calculate mean/median/ stdev values for a given mnemonic. The method used depends on the type of telemetry.

Parameters:

mnemonic_instancejwql.edb.engineering_database.EdbMnemonic

EdbMnemonic instance containing the telemetry data to be averaged.

telemetry_typestr

Type of telemetry. Examples include “daily”, “every_change”, “all”. These values come from the top-level json file that lists the mnemonics to be monitored.

Returns:

mnemonic_instancejwql.edb.engineering_database.EdbMnemonic

Modified EdbMnemonic instance with the “mean”, “median”, and “stdev” attributes populated.

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.define_options(parser=None, usage=None, conflict_handler='resolve')

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.empty_edb_instance(name, beginning, ending, meta={}, info={})

Create an EdbMnemonic instance with an empty data table

Parameters:

namestr

Name of mnemonic to attach to the empty EdbMnemonic instance

beginningdatetime.datetime

Starting time value associated with empty instance

endingdatetime.datetime

Ending time value associated with empty instance

metadict

Meta data dictionary to attach to meta attribute

infodict

Info dictionary to attach to info attribute

Returns:

varjwql.edb.engineering_database.EdbMnemonic

Empty instance of EdbMnemonic

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.ensure_list(var)

Be sure that var is a list. If not, make it one.

Parameters:

varlist or str or float or int

Variable to be checked

Returns:

varlist

var, translated into a list if necessary

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.organize_every_change(mnemonic)

Given an EdbMnemonic instance containing every_change data, organize the information such that there are single 1d arrays of data and dates for each of the dependency values. This will make plotting and saving in the database much more straight forward. Note that this is intended to be run on an EdbMnenonic instance that has come out of multiday_mnemonic_query, so that the data table contains averaged values. In this case, the data in the data table will line up with the values given in the every_change_values attribute.

Parameters:

mnemonicjwql.edb.engineering_database.EdbMnemonic

Object containing all data

Returns:

all_datadict

Dictionary of organized results. Keys are the dependency values, and values are tuples. The first element of each tuple is a list of dates, the second element is a list of data values, and the third is a the sigma-clipped mean value of the data.

jwql.instrument_monitors.common_monitors.edb_telemetry_monitor.plot_every_change_data(data, mnem_name, units, show_plot=False, savefig=True, out_dir='./', nominal_value=None, yellow_limits=None, red_limits=None, xrange=(None, None), yrange=(None, None), title=None, return_components=True, return_fig=False, minimal_start=None, minimal_end=None)

Create a plot for mnemonics where we want to see the behavior within each change

Parameters:

datacollections.defaultdict

Dictionary containing every_change data to be plotted. Keys should be the values of the dependency mnemonic, and values should be 3-tuples (list of datetimes, list of data, mean value)

mnem_namestr

Name of the mnemonic being plotted. This will be used to generate a filename in the case where the figure is saved.

unitsastropy.units.unit

Units associated with the data. This will be used as the y-axis label in the plot

show_plotbool

If True, show plot on screen rather than returning div and script

savefigbool

If True, file is saved to html file

out_dirstr

Directory into which the html file is saved

nominal_valuefloat

Expected or nominal value for the telemetry. If provided, a horizontal dashed line at this value will be added.

yellow_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered non-nominal. If provided, the area of the plot between these two values will be given a green background, and that outside of these limits will have a yellow background.

red_limitslist

2-element list giving the lower and upper limits outside of which the telemetry value is considered worse than in the yellow region. If provided, the area of the plot outside of these two values will have a red background.

xrangetuple

Tuple of min, max datetime values to use as the plot range in the x direction.

yrangetuple

Tuple of min, max datetime values to use as the plot range in the y direction.

titlestr

Will be used as the plot title. If None, mnem_name will be used as the title

return_componentsbool

If True, the components (script, div) of the figure will be returned

return_figbool

If True, the Bokeh figure will be returned

minimal_startdatetime.datetime

In the case where the data to be plotted consists of no or only one point, use this as the earliest date in the plot

minimal_enddatetime.datetime

In the case where the data to be plotted consists of no or only one point, use this as the latest date in the plot

Returns:

objlist or bokeh.plotting.figure

If return_components is True, returned object will be a list of [div, script] If return_figure is True, a bokeh.plotting.figure will be returned

readnoise_monitor.py

This module contains code for the readnoise monitor, which monitors the readnoise levels in dark exposures as well as the accuracy of the pipeline readnoise reference files over time.

For each instrument, the readnoise, technically the correlated double sampling (CDS) noise, is found by calculating the standard deviation through a stack of consecutive frame differences in each dark exposure. The sigma-clipped mean and standard deviation in each of these readnoise images, as well as histogram distributions, are recorded in the <Instrument>ReadnoiseStats database table.

Next, each of these readnoise images are differenced with the current pipeline readnoise reference file to identify the need for new reference files. A histogram distribution of these difference images, as well as the sigma-clipped mean and standard deviation, are recorded in the <Instrument>ReadnoiseStats database table. A png version of these difference images is also saved for visual inspection.

Author

• Ben Sunnquist

Use

This module can be used from the command line as such:

python readnoise_monitor.py

class jwql.instrument_monitors.common_monitors.readnoise_monitor.Readnoise

Class for executing the readnoise monitor.

This class will search for new dark current files in the file system for each instrument and will run the monitor on these files. The monitor will create a readnoise image for each of the new dark files. It will then perform statistical measurements on these readnoise images, as well as their differences with the current pipeline readnoise reference file, in order to monitor the readnoise levels over time as well as ensure the pipeline readnoise reference file is sufficiently capturing the current readnoise behavior. Results are all saved to database tables.

Attributes:

output_dirstr

Path into which outputs will be placed.

working_dirstr

Path into which new dark files will be copied to be worked on.

query_startfloat

MJD start date to use for querying MAST.

query_endfloat

MJD end date to use for querying MAST.

instrumentstr

Name of instrument used to collect the dark current data.

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL).

determine_pipeline_steps()

Determines the necessary JWST pipelines steps to run on a given dark file.

Returns:

pipeline_stepscollections.OrderedDict

The pipeline steps to run.

file_exists_in_database(filename)

Checks if an entry for filename exists in the readnoise stats database.

Parameters:

filenamestr

The full path to the uncal filename.

Returns:

file_existsbool

True if filename exists in the readnoise stats database.

get_amp_stats(image, amps)

Calculates the sigma-clipped mean and stddev, as well as the histogram stats in the input image for each amplifier.

Parameters:

imagenumpy.ndarray

2D array on which to calculate statistics.

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)]

Returns:

amp_statsdict

Contains the image statistics for each amp.

get_metadata(filename)

Collect basic metadata from a fits file.

Parameters:

filenamestr

Name of fits file to examine.

identify_tables()

Determine which database tables to use for a run of the readnoise monitor.

image_to_png(image, outname)

Outputs an image array into a png file.

Parameters:

imagenumpy.ndarray

2D image array.

outnamestr

The name given to the output png file.

Returns:

output_filenamestr

The full path to the output png file.

make_crds_parameter_dict()

Construct a paramter dictionary to be used for querying CRDS for the current reffiles in use by the JWST pipeline.

Returns:

parametersdict

Dictionary of parameters, in the format expected by CRDS.

make_histogram(data)

Creates a histogram of the input data and returns the bin centers and the counts in each bin.

Parameters:

datanumpy.ndarray

The input data.

Returns:

countsnumpy.ndarray

The counts in each histogram bin.

bin_centersnumpy.ndarray

The histogram bin centers.

make_readnoise_image(data)

Calculates the readnoise for the given input dark current ramp.

Parameters:

datanumpy.ndarray

The input ramp data. The data shape is assumed to be a 4D array in DMS format (integration, group, y, x).

Returns:

readnoisenumpy.ndarray

The 2D readnoise image.

most_recent_search()

Query the query history database and return the information on the most recent query for the given aperture_name where the readnoise monitor was executed.

Returns:

query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the readnoise monitor was run.

process(file_list)

The main method for processing darks. See module docstrings for further details.

Parameters:

file_listlist

List of filenames (including full paths) to the dark current files.

run(**kwargs)

Caller.