jwql_monitors

generate_preview_images.py

Generate preview images for all files in the jwql filesystem.

Execution of this script will generate preview images and thumbnail images for each file in the jwql filesystem. Preview images have axes labels, titles, and colorbars, wheras thumbnail images are smaller and contain no labels. Images are saved into the preview_image_filesystem and thumbnail_filesystem, organized by subdirectories pertaining to the program_id in the filenames.

Authors

• Matthew Bourque

• Bryan Hilbert

Use

This script is intended to be executed as such:

python generate_preview_images.py

jwql.jwql_monitors.generate_preview_images.array_coordinates(channelmod, detector_list, lowerleft_list)

Create an appropriately sized numpy array to contain the mosaic image given the channel and module of the data.

Parameters:

channelmodstr

Indicator of the NIRCam channel/module of the data. Options are: LW - for longwave channel data SWA - for shortwave A module only (4 detectors) data SWB - for shortwave B module only (4 detectors) data SW - for shortwave both module data (8 detectors)

detector_listlist

List of detectors used in data to be simulated

lowerleft_listlist

Each element is a tuple giving the (x, y) coordinates corresponding to the lower left corner of the aperture within the full frame detector. These values come from the SUBSTRT1 and 2 values in the file headers.

Returns:

xdimint

Length of the output array needed to contain all detectors’ data

ydimint

Height of the output array needed to contain all detectors’ data

module_lowerleftsdict

Dictionary giving the (x, y) coordinate in the coordinate system of the full module(s) where the lower left corner of the data from a given detector will be placed. (e.g. NRCA1: (1888, 1888) means that the data from detector NRCA1 should be placed into [1888: 1888+y_dim_of_data, 1888: 1888+x_dim_of_data] within the final array (which has total dimensions of (xdim, ydim)

jwql.jwql_monitors.generate_preview_images.check_existence(file_list, outdir)

Given a list of fits files, determine if a preview image has already been created in outdir.

Parameters:

file_listlist

List of fits filenames from which preview image will be generated

outdirstr

Directory that will contain the preview image if it exists

Returns:

existsbool

True if preview image exists, False if it does not

jwql.jwql_monitors.generate_preview_images.create_dq_array(xd, yd, mosaic, module)

Create DQ array that goes with the mosaic image. Set unpopulated pixels to be skipped in preview image scaling. Same for the reference pixels for all detectors

Parameters:

xdint

X-coordinate dimension of the DQ array

ydint

Y-coordinate dimension of the DQ array

mosaicobj

2D numpy array containing the mosaic image

modulestr

Module used for mosaic. Options are LW,`` SW``, SWA, SWB

Returns:

dqobj

2D numpy array containing the DQ array. Pixels that are True are considered science pixels and are used when scaling the preview image. Pixels that are False are skipped.

jwql.jwql_monitors.generate_preview_images.create_dummy_filename(filelist)

Create a dummy filename indicating the detectors used to create the mosaic. Check the list of detectors used to determine the proper text to substitute into the initial filename.

Parameters:

filelistlist

List of filenames containing the data used to create the mosaic. It is assumed these filenames follow JWST filenaming conventions.

Returns:

dummy_namestr

The first filename in filelist is modified, such that the detector name is replaced with text indicating the source of the mosaic data.

jwql.jwql_monitors.generate_preview_images.create_mosaic(filenames)

If an exposure comprises data from multiple detectors read in all the appropriate files and create a mosaic so that the preview image will show all the data together.

Parameters:

filenameslist

List of filenames to be combined into a mosaic

Returns:

mosaic_filenamestr

Name of fits file containing the mosaicked data

jwql.jwql_monitors.generate_preview_images.define_options(parser=None, usage=None, conflict_handler='resolve')

jwql.jwql_monitors.generate_preview_images.detector_check(detector_list, search_string)

Search a given list of detector names for the provided regular expression sting.

Parameters:

detector_listlist

List of detector names (e.g. NRCA5)

search_stringstr

Regular expression string to use for search

Returns:

totalint

Number of detectors in detector_list that match search_string

jwql.jwql_monitors.generate_preview_images.find_data_channel(detectors)

Using a list of detectors, identify the channel(s) that the data are from.

Parameters:

detectorslist

List of detector names

Returns:

channelstr

Identifier noting which channels the given detectors are in. Can be SWA for shortwave, module A only, SWB for shortwave, module B only, SW, for shortwave modules A and B, and LW for longwave.

jwql.jwql_monitors.generate_preview_images.generate_preview_images(overwrite)

The main function of the generate_preview_image module. See module docstring for further details.

Parameters:

overwritebool

If True, any existing preview images and thumbnails are overwritten

jwql.jwql_monitors.generate_preview_images.get_base_output_name(filename_dict)

Returns the base output name used for preview images and thumbnails.

Parameters:

filename_dictdict

A dictionary containing parsed filename parts via filename_parser

Returns:

base_output_namestr

The base output name, e.g. jw96090001002_03101_00001_nrca2_rate

jwql.jwql_monitors.generate_preview_images.group_filenames(filenames)

Given a list of JWST filenames, group together files from the same exposure. These files will share the same program_id, observation, visit, visit_group, parallel_seq_id, activity, exposure, and suffix. Only the detector will be different. Currently only NIRCam files for a given exposure will be grouped together. For other instruments multiple files for a given exposure will be kept separate from one another and no mosaic will be made. Stage 3 files will remain as individual files, and will not be grouped together with any other files.

Parameters:

filenameslist

list of filenames

Returns:

groupedlist

grouped list of filenames where each element is a list and contains the names of filenames with matching exposure information.

jwql.jwql_monitors.generate_preview_images.process_program(program, overwrite)

Generate preview images and thumbnails for the given program.

Parameters:

programstr

The program identifier (e.g. 88600)

overwritebool

If False, skip over preview images/thumbnails that already exist. Only create images that do not currenlty exist. If True, create preview_images and thumbnails for all input files, regardless of whether the images already exist.

Returns:

preview_image_fileslist

List of preview image filenames

thumbnail_fileslist

List of thumbnail image filenames

jwql.jwql_monitors.generate_preview_images.protected_code(overwrite)

Protected code ensures only 1 instance of module will run at any given time

Parameters:

overwritebool

If True, any existing preview images and thumbnails are overwritten

jwql.jwql_monitors.generate_preview_images.update_listfile(filename, file_list, filetype)

Add a list of files to a text file. Designed to add new files to the file containing the list of all preview images and the file containing the list of all thumbnail images.

Parameters:

filenamestr

Name, including path, of the file to be amended/created

file_listlist

List of filenames to be added to filename

filetypestr

Descriptor of the contents of the file being amended. Used only for the logging statement

generate_proposal_thumbnails.py

Generate thumbnails for each proposal to be used for display in the archive page of the web app.

In the web app, each instrument has its own ‘archive’ page where users may view JWST images for the particular instrument. The page displays buttons for each proposal with a thumbnail image from each particular proposal. This script goes through the thumbnail filesystem and creates the thumbnail to display based on the first existing rate file. The thumbnail is saved as <proposal_id>.thumb.

Authors

• Matthew Bourque

Use

This script is intended to be executed as such:

python generate_proposal_thumbnails.py

jwql.jwql_monitors.generate_proposal_thumbnails.generate_proposal_thumbnails()

The main function of the generate_proposal_thumbnails module. See module docstring for further details.

jwql.jwql_monitors.generate_proposal_thumbnails.protected_code()

Protected code ensures only 1 instance of module will run at any given time

monitor_cron_jobs.py

This module monitors the status of the jwql monitors via their log files. Basic results (e.g. success, failure) are collected and placed in a bokeh table for display on the web app.

Authors

• Bryan Hilbert

Use

This module can be executed as such:

from jwql.jwql_monitors import monitor_cron_jobs
monitor_cron_jobs.status()

Dependencies

The user must have a configuration file named config.json placed in the jwql directory.

jwql.jwql_monitors.monitor_cron_jobs.create_table(status_dict)

Create interactive bokeh table containing the logfile status results.

Parameters:

status_dictdict

Nested dictionary with status results from all logfiles

jwql.jwql_monitors.monitor_cron_jobs.find_latest(logfiles)

Given a list of log files in a directory, identify the most recent. The way that jwql.utils.logging_functions.make_log_file is set up, log files for all monitors are guaranteed to be the name of the monitor followed by the datetime that they were run, so we should be able to simply sort the filenames and the last will be the most recent.

Parameters:

logfileslist

List of logfiles in the directory

Returns:

lateststr

Filename of the most recent file

latest_timefloat

Time associated with the most recent log file

jwql.jwql_monitors.monitor_cron_jobs.get_cadence(filenames)

Calculate the cadence of the log files in a given directory. Use timestamps

Parameters:

filenameslist

List of log files to examine

Returns:

mean_deltafloat

Mean time in seconds between the appearance of consecutive log files

stdev_deltafloat

Standard deviation in seconds between the appearance of consecutive log files

jwql.jwql_monitors.monitor_cron_jobs.missing_file_check(avg_time_between, uncertainty, latest_file)

Given the name of the most recent log file, along with the historical average time between files and the stdev of the time between files, determine whether we expect a more recent log file than the file given. This could hint at a problem with the cron job used to create the log files.

Parameters:

avg_time_betweenfloat

Average number of seconds between log files

uncertaintyfloat

Standard deviation of the number of seconds between log files

latest_filestr

Name of the most recent log file

Returns:

latebool

True = We expect a more recent file than that given False = It is reasonable that the file given is the most recent

jwql.jwql_monitors.monitor_cron_jobs.protected_code()

Protected code ensures only 1 instance of module will run at any given time

jwql.jwql_monitors.monitor_cron_jobs.status(production_mode=True)

Main function: determine the status of the instrument montiors by examining log files.

Parameters:

production_modebool

If True, look in the main log directory. If False, look in the dev log file directory.

Returns:

logfile_statusdict

Nested dictionary containing the status for all monitors. Top level keys include all monitors. Within a given monitor, the value is a dictionary containing ‘missing_file’ and ‘status’ keys. ‘missing_file’ is a boolean describing whether or not there is a suspected missing log file based on the timestamps of the existing files. ‘status’ is a string that is either ‘success’ or ‘failure’.

jwql.jwql_monitors.monitor_cron_jobs.success_check(filename)

Parse the given log file and check whether the script execution was successful or not

Parameters:

filenamestr

Name of the log file to parse

Returns:

executionstr

success or failure

monitor_filesystem.py

This module monitors and gather statistics of the filesystem and central storage area that hosts data for the jwql application. This will answer questions such as the total number of files, how much disk space is being used, and then plot these values over time.

Authors

• Misty Cracraft

• Sara Ogaz

• Matthew Bourque

• Bryan Hilbert

Use

This module is intended to be executed from the command line:

python monitor_filesystem.py

The user must have a config.json file in the jwql directory with the following keys:

• filesystem - The path to the filesystem

• outputs - The path to where the output plots will be

  written

Dependencies

The user must have a configuration file named config.json placed in the jwql directory.

jwql.jwql_monitors.monitor_filesystem.files_per_filter()

Querying MAST (rather than looping through the filesystem), determine how many files use each filter for each instrument. Note that thiw function takes a long time (~minutes per filter) to execute.

Returns:

n_obsdict

Dictionary with filter names as keys, and values of the number of Observations that use that particular filter.

jwql.jwql_monitors.monitor_filesystem.gather_statistics(general_results_dict, instrument_results_dict)

Walks the filesytem to gather various statistics to eventually store in the database

Parameters:

general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

Returns:

general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

jwql.jwql_monitors.monitor_filesystem.get_area_stats(central_storage_dict)

Gathers used and available df-style stats on the selected area.

Parameters:

central_storage_dictdict

A dictionary for the central_storage database table

Returns:

central_storage_dictdict

A dictionary for the central_storage database table

jwql.jwql_monitors.monitor_filesystem.get_global_filesystem_stats(general_results_dict)

Gathers used and available df-style stats on the entire filesystem. (Not just directory titled filesystem.)

Parameters:

general_results_dictdict

A dictionary for the filesystem_general database table

Returns:

general_results_dictdict

A dictionary for the filesystem_general database table

jwql.jwql_monitors.monitor_filesystem.get_observation_characteristics()

Query MAST and count the number of observations that make use of each filter/pupil pair for each instrument.

Returns:

n_obsdict

Dictionary with instrument names as the top level keys, and lists of 2-tuples as values. Each tuple contains filter/pupil string and the number of observations that use that filter/pupil.

jwql.jwql_monitors.monitor_filesystem.initialize_results_dicts()

Initializes dictionaries that will hold filesystem statistics

Returns:

general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table

central_storage_dictdict

A dictionary for the central_storage database table

jwql.jwql_monitors.monitor_filesystem.monitor_filesystem()

Tabulates the inventory of the JWST filesystem, saving statistics to database tables, and generates plots.

jwql.jwql_monitors.monitor_filesystem.protected_code()

Protected code ensures only 1 instance of module will run at any given time

jwql.jwql_monitors.monitor_filesystem.update_central_store_database(central_storage_dict)

Updates the CentralStore database table with info on disk space

Parameters:

central_storage_dictdict

A dictionary for the central_storage database table

jwql.jwql_monitors.monitor_filesystem.update_characteristics_database(char_info)

Updates the filesystem_characteristics database table.

Parameters:

char_infodict

A dictionary of characteristic information. Keys are instrument names, and values are lists of tuples. Each tuple is composed of a filter/pupil string and a count for the number of observations using that filter/pupil.

jwql.jwql_monitors.monitor_filesystem.update_database(general_results_dict, instrument_results_dict, central_storage_dict)

Updates the filesystem_general and filesystem_instrument database tables.

Parameters:

general_results_dictdict

A dictionary for the filesystem_general database table

instrument_results_dictdict

A dictionary for the filesystem_instrument database table