jwql_monitors

generate_preview_images.py

Generate preview images for all files in the jwql filesystem.

Execution of this script will generate preview images and thumbnail images for each file in the jwql filesystem. Preview images have axes labels, titles, and colorbars, wheras thumbnail images are smaller and contain no labels. Images are saved into the preview_image_filesystem and thumbnail_filesystem, organized by subdirectories pertaining to the program_id in the filenames.

Authors

  • Matthew Bourque

  • Bryan Hilbert

Use

This script is intended to be executed as such:

python generate_preview_images.py
jwql.jwql_monitors.generate_preview_images.array_coordinates(channelmod, detector_list, lowerleft_list)[source]

Create an appropriately sized numpy array to contain the mosaic image given the channel and module of the data.

Parameters:
channelmodstr

Indicator of the NIRCam channel/module of the data. Options are: LW - for longwave channel data SWA - for shortwave A module only (4 detectors) data SWB - for shortwave B module only (4 detectors) data SW - for shortwave both module data (8 detectors)

detector_listlist

List of detectors used in data to be simulated

lowerleft_listlist

Each element is a tuple giving the (x, y) coordinates corresponding to the lower left corner of the aperture within the full frame detector. These values come from the SUBSTRT1 and 2 values in the file headers.

Returns:
xdimint

Length of the output array needed to contain all detectors’ data

ydimint

Height of the output array needed to contain all detectors’ data

module_lowerleftsdict

Dictionary giving the (x, y) coordinate in the coordinate system of the full module(s) where the lower left corner of the data from a given detector will be placed. (e.g. NRCA1: (1888, 1888) means that the data from detector NRCA1 should be placed into [1888: 1888+y_dim_of_data, 1888: 1888+x_dim_of_data] within the final array (which has total dimensions of (xdim, ydim)

jwql.jwql_monitors.generate_preview_images.check_existence(file_list, outdir)[source]

Given a list of fits files, determine if a preview image has already been created in outdir.

Parameters:
file_listlist

List of fits filenames from which preview image will be generated

outdirstr

Directory that will contain the preview image if it exists

Returns:
existsbool

True if preview image exists, False if it does not

jwql.jwql_monitors.generate_preview_images.create_dq_array(xd, yd, mosaic, module)[source]

Create DQ array that goes with the mosaic image. Set unpopulated pixels to be skipped in preview image scaling. Same for the reference pixels for all detectors

Parameters:
xdint

X-coordinate dimension of the DQ array

ydint

Y-coordinate dimension of the DQ array

mosaicobj

2D numpy array containing the mosaic image

modulestr

Module used for mosaic. Options are LW,`` SW``, SWA, SWB

Returns:
dqobj

2D numpy array containing the DQ array. Pixels that are True are considered science pixels and are used when scaling the preview image. Pixels that are False are skipped.

jwql.jwql_monitors.generate_preview_images.create_dummy_filename(filelist)[source]

Create a dummy filename indicating the detectors used to create the mosaic. Check the list of detectors used to determine the proper text to substitute into the initial filename.

Parameters:
filelistlist

List of filenames containing the data used to create the mosaic. It is assumed these filenames follow JWST filenaming conventions.

Returns:
dummy_namestr

The first filename in filelist is modified, such that the detector name is replaced with text indicating the source of the mosaic data.

jwql.jwql_monitors.generate_preview_images.create_mosaic(filenames)[source]

If an exposure comprises data from multiple detectors read in all the appropriate files and create a mosaic so that the preview image will show all the data together.

Parameters:
filenameslist

List of filenames to be combined into a mosaic

Returns:
mosaic_filenamestr

Name of fits file containing the mosaicked data

jwql.jwql_monitors.generate_preview_images.detector_check(detector_list, search_string)[source]

Search a given list of detector names for the provided regular expression sting.

Parameters:
detector_listlist

List of detector names (e.g. NRCA5)

search_stringstr

Regular expression string to use for search

Returns:
totalint

Number of detectors in detector_list that match search_string

jwql.jwql_monitors.generate_preview_images.find_data_channel(detectors)[source]

Using a list of detectors, identify the channel(s) that the data are from.

Parameters:
detectorslist

List of detector names

Returns:
channelstr

Identifier noting which channels the given detectors are in. Can be SWA for shortwave, module A only, SWB for shortwave, module B only, SW, for shortwave modules A and B, and LW for longwave.

jwql.jwql_monitors.generate_preview_images.generate_preview_images()[source]

The main function of the generate_preview_image module. See module docstring for further details.

jwql.jwql_monitors.generate_preview_images.get_base_output_name(filename_dict)[source]

Returns the base output name used for preview images and thumbnails.

Parameters:
filename_dictdict

A dictionary containing parsed filename parts via filename_parser

Returns:
base_output_namestr

The base output name, e.g. jw96090001002_03101_00001_nrca2_rate

jwql.jwql_monitors.generate_preview_images.group_filenames(filenames)[source]

Given a list of JWST filenames, group together files from the same exposure. These files will share the same program_id, observation, visit, visit_group, parallel_seq_id, activity, exposure, and suffix. Only the detector will be different. Currently only NIRCam files for a given exposure will be grouped together. For other instruments multiple files for a given exposure will be kept separate from one another and no mosaic will be made. Stage 3 files will remain as individual files, and will not be grouped together with any other files.

Parameters:
filenameslist

list of filenames

Returns:
groupedlist

grouped list of filenames where each element is a list and contains the names of filenames with matching exposure information.

jwql.jwql_monitors.generate_preview_images.process_program(program)[source]

Generate preview images and thumbnails for the given program.

Parameters:
programstr

The program identifier (e.g. 88600)

generate_proposal_thumbnails.py

Generate thumbnails for each proposal to be used for display in the archive page of the web app.

In the web app, each instrument has its own ‘archive’ page where users may view JWST images for the particular instrument. The page displays buttons for each proposal with a thumbnail image from each particular proposal. This script goes through the thumbnail filesystem and creates the thumbnail to display based on the first existing rate file. The thumbnail is saved as <proposal_id>.thumb.

Authors

  • Matthew Bourque

Use

This script is intended to be executed as such:

python generate_proposal_thumbnails.py
jwql.jwql_monitors.generate_proposal_thumbnails.generate_proposal_thumbnails()[source]

The main function of the generate_proposal_thumbnails module. See module docstring for further details.

monitor_cron_jobs.py

This module monitors the status of the jwql monitors via their log files. Basic results (e.g. success, failure) are collected and placed in a bokeh table for display on the web app.

Authors

  • Bryan Hilbert

Use

This module can be executed as such:

from jwql.jwql_monitors import monitor_cron_jobs
monitor_cron_jobs.status()

Dependencies

The user must have a configuration file named config.json placed in the jwql directory.

jwql.jwql_monitors.monitor_cron_jobs.create_table(status_dict)[source]

Create interactive bokeh table containing the logfile status results.

Parameters:
status_dictdict

Nested dictionary with status results from all logfiles

jwql.jwql_monitors.monitor_cron_jobs.find_latest(logfiles)[source]

Given a list of log files in a directory, identify the most recent. The way that jwql.utils.logging_functions.make_log_file is set up, log files for all monitors are guaranteed to be the name of the monitor followed by the datetime that they were run, so we should be able to simply sort the filenames and the last will be the most recent.

Parameters:
logfileslist

List of logfiles in the directory

Returns:
lateststr

Filename of the most recent file

latest_timefloat

Time associated with the most recent log file

jwql.jwql_monitors.monitor_cron_jobs.get_cadence(filenames)[source]

Calculate the cadence of the log files in a given directory. Use timestamps

Parameters:
filenameslist

List of log files to examine

Returns:
mean_deltafloat

Mean time in seconds between the appearance of consecutive log files

stdev_deltafloat

Standard deviation in seconds between the appearance of consecutive log files

jwql.jwql_monitors.monitor_cron_jobs.missing_file_check(avg_time_between, uncertainty, latest_file)[source]

Given the name of the most recent log file, along with the historical average time between files and the stdev of the time between files, determine whether we expect a more recent log file than the file given. This could hint at a problem with the cron job used to create the log files.

Parameters:
avg_time_betweenfloat

Average number of seconds between log files

uncertaintyfloat

Standard deviation of the number of seconds between log files

latest_filestr

Name of the most recent log file

Returns:
latebool

True = We expect a more recent file than that given False = It is reasonable that the file given is the most recent

jwql.jwql_monitors.monitor_cron_jobs.status(production_mode=True)[source]

Main function: determine the status of the instrument montiors by examining log files.

Parameters:
production_modebool

If True, look in the main log directory. If False, look in the dev log file directory.

Returns:
logfile_statusdict

Nested dictionary containing the status for all monitors. Top level keys include all monitors. Within a given monitor, the value is a dictionary containing ‘missing_file’ and ‘status’ keys. ‘missing_file’ is a boolean describing whether or not there is a suspected missing log file based on the timestamps of the existing files. ‘status’ is a string that is either ‘success’ or ‘failure’.

jwql.jwql_monitors.monitor_cron_jobs.success_check(filename)[source]

Parse the given log file and check whether the script execution was successful or not

Parameters:
filenamestr

Name of the log file to parse

Returns:
executionstr

success or failure

monitor_filesystem.py

This module monitors and gather statistics of the filesystem and central storage area that hosts data for the jwql application. This will answer questions such as the total number of files, how much disk space is being used, and then plot these values over time.

Authors

  • Misty Cracraft

  • Sara Ogaz

  • Matthew Bourque

Use

This module is intended to be executed from the command line:

python monitor_filesystem.py

The user must have a config.json file in the jwql directory with the following keys:

  • filesystem - The path to the filesystem

  • outputs - The path to where the output plots will be

    written

Dependencies

The user must have a configuration file named config.json placed in the jwql directory.

jwql.jwql_monitors.monitor_filesystem.gather_statistics(general_results_dict, instrument_results_dict)[source]

Walks the filesytem to gather various statistics to eventually store in the database

Parameters:
general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

Returns:
general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

jwql.jwql_monitors.monitor_filesystem.get_area_stats(central_storage_dict)[source]

Gathers used and available df-style stats on the selected area.

Parameters:
central_storage_dictdict

A dictionary for the central_storage database table

Returns:
central_storage_dictdict

A dictionary for the central_storage database table

jwql.jwql_monitors.monitor_filesystem.get_global_filesystem_stats(general_results_dict)[source]

Gathers used and available df-style stats on the entire filesystem. (Not just directory titled filesystem.)

Parameters:
general_results_dictdict

A dictionary for the filesystem_general database table

Returns:
general_results_dictdict

A dictionary for the filesystem_general database table

jwql.jwql_monitors.monitor_filesystem.initialize_results_dicts()[source]

Initializes dictionaries that will hold filesystem statistics

Returns:
general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

central_storage_dictdict

A dictionary for the central_storage database table

jwql.jwql_monitors.monitor_filesystem.monitor_filesystem()[source]

Tabulates the inventory of the JWST filesystem, saving statistics to database tables, and generates plots.

jwql.jwql_monitors.monitor_filesystem.plot_by_filetype(plot_type, instrument)[source]

Plot count or size by filetype versus date for the given instrument, or all instruments.

Parameters:
plot_typestr

Which data to plot. Either count or size.

instrumentstr

The instrument to plot for. Can be a valid JWST instrument or all to plot across all instruments.

Returns:
plotbokeh.plotting.figure.Figure object

bokeh plot of total file counts versus date

jwql.jwql_monitors.monitor_filesystem.plot_central_store_dirs()[source]

Plot central store sizes (size, used, available) versus date

Returns:
plotbokeh.plotting.figure.Figure object

bokeh plot of total directory size versus date

jwql.jwql_monitors.monitor_filesystem.plot_filesystem_size()[source]

Plot filesystem sizes (size, used, available) versus date

Returns:
plotbokeh.plotting.figure.Figure object

bokeh plot of total file counts versus date

jwql.jwql_monitors.monitor_filesystem.plot_filesystem_stats()[source]

Plot various filesystem statistics using bokeh and save them to the output directory.

jwql.jwql_monitors.monitor_filesystem.plot_total_file_counts()[source]

Plot total file counts versus date

Returns:
plotbokeh.plotting.figure.Figure object

bokeh plot of total file counts versus date

jwql.jwql_monitors.monitor_filesystem.update_database(general_results_dict, instrument_results_dict, central_storage_dict)[source]

Updates the filesystem_general and filesystem_instrument database tables.

Parameters:
general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

central_storage_dictdict

A dictionary for the central_storage database table

monitor_mast.py

This module is home to a suite of MAST queries that gather bulk properties of available JWST data for JWQL.

Authors

Joe Filippazzo

Use

To get an inventory of all JWST files do:

from jwql.jwql_monitors import monitor_mast
inventory, keywords = monitor_mast.jwst_inventory()
jwql.jwql_monitors.monitor_mast.instrument_inventory(instrument, dataproduct=['IMAGE', 'SPECTRUM', 'SED', 'TIMESERIES', 'VISIBILITY', 'EVENTLIST', 'CUBE', 'CATALOG', 'ENGINEERING', 'NULL'], add_filters=None, add_requests=None, caom=False, return_data=False)[source]

Get the counts for a given instrument and data product

Parameters:
instrument: str

The instrument name, i.e. one of [‘niriss’,’nircam’,’nirspec’, ‘miri’,’fgs’]

dataproduct: sequence, str

The type of data product to search

add_filters: dict

The (‘paramName’:’values’) pairs to include in the ‘filters’ argument of the request e.g. add_filters = {‘filter’:’GR150R’}

add_requests: dict

The (‘request’:’value’) pairs to include in the request e.g. add_requests = {‘pagesize’:1, ‘page’:1}

caom: bool

Query CAOM service

return_data: bool

Return the actual data instead of counts only

Returns:
int, dict

The number of database records that satisfy the search criteria or a dictionary of the data if return_data=True

jwql.jwql_monitors.monitor_mast.instrument_keywords(instrument, caom=False)[source]

Get the keywords for a given instrument service

Parameters:
instrument: str

The instrument name, i.e. one of [‘niriss’,’nircam’,’nirspec’, ‘miri’,’fgs’]

caom: bool

Query CAOM service

Returns:
pd.DataFrame

A DataFrame of the keywords

jwql.jwql_monitors.monitor_mast.jwst_inventory(instruments=['fgs', 'miri', 'nircam', 'niriss', 'nirspec'], dataproducts=['image', 'spectrum', 'cube'], caom=False, plot=False)[source]

Gather a full inventory of all JWST data in each instrument service by instrument/dtype

Parameters:
instruments: sequence

The list of instruments to count

dataproducts: sequence

The types of dataproducts to count

caom: bool

Query CAOM service

plot: bool

Return a pie chart of the data

Returns:
astropy.table.table.Table

The table of record counts for each instrument and mode

jwql.jwql_monitors.monitor_mast.monitor_mast()[source]

Tabulates the inventory of all JWST data products in the MAST archive and generates plots.