utils¶

calculations.py¶

Various math-related functions used by the jwql instrument monitors.

Authors¶

Bryan Hilbert

Use¶

This module can be imported as such:

from jwql.utils import calculations
mean_val, stdev_val = calculations.mean_stdev(image, sigma_threshold=4)

jwql.utils.calculations.double_gaussian(x, amp1, peak1, sigma1, amp2, peak2, sigma2)[source]¶

Equate two Gaussians

Parameters:

xnumpy.ndarray: 1D array of x values to be fit
paramslist: Gaussian coefficients [amplitude1, peak1, stdev1, amplitude2, peak2, stdev2]

jwql.utils.calculations.double_gaussian_fit(x_values, y_values, input_params)[source]¶

Fit two Gaussians to the given array

Parameters:

x_valuesnumpy.ndarray: 1D array of x values to be fit
y_valuesnumpy.ndarray: 1D array of y values to be fit
input_paramslist: Initial guesses for Gaussian coefficients [amplitude1, peak1, stdev1, amplitude2, peak2, stdev2]

Returns:

paramslist: Fitted parameter values
sigmanumpy.ndarray: Uncertainties on the parameters

jwql.utils.calculations.gaussian1d_fit(x_values, y_values, params)[source]¶

Fit 1D Gaussian to an array. Designed around fitting to histogram of pixel values.

Parameters:

x_valuesnumpy.ndarray: 1D array of x values to be fit
y_valuesnumpy.ndarray: 1D array of y values to be fit

Returns:

amplitudetup: Tuple of the best fit Gaussian amplitude and uncertainty
peaktup: Tuple of the best fit Gaussian peak position and uncertainty
widthtup: Tuple of the best fit Gaussian width and uncertainty

jwql.utils.calculations.mean_image(cube, sigma_threshold=3)[source]¶

Combine a stack of 2D images into a mean slope image, using sigma-clipping on a pixel-by-pixel basis

Parameters:

cubenumpy.ndarray: 3D array containing a stack of 2D images
sigma_thresholdint: Number of sigma to use when sigma-clipping values in each pixel

Returns:

mean_imagenumpy.ndarray: 2D sigma-clipped mean image
stdev_imagenumpy.ndarray: 2D sigma-clipped standard deviation image

jwql.utils.calculations.mean_stdev(image, sigma_threshold=3)[source]¶

Calculate the sigma-clipped mean and stdev of an input array

Parameters:

imagenumpy.ndarray: Array of which to calculate statistics
sigma_thresholdfloat: Number of sigma to use when sigma-clipping

Returns:

mean_valuefloat: Sigma-clipped mean of image
stdev_valuefloat: Sigma-clipped standard deviation of image

constants.py¶

Globally defined and used variables for the jwql project.

Authors¶

Johannes Sahlmann

Matthew Bourque

Bryan Hilbert

Ben Sunnquist

Teagan King

Mike Engesser

Maria Pena-Guerrero

Rachel Cooper

Brad Sappington

Use¶

This variables within this module are intended to be directly imported, e.g.:
from jwql.utils.constants import JWST_INSTRUMENT_NAMES

References¶

Many variables were transferred from an earlier version of utils.py

class jwql.utils.constants.QueryConfigKeys[source]¶

ANOMALIES = 'ANOMALIES'¶

APERTURES = 'APERTURES'¶

DATE_RANGE = 'DATE_RANGE'¶

DETECTORS = 'DETECTORS'¶

EXP_TYPES = 'EXP_TYPES'¶

FILTERS = 'FILTERS'¶

GRATINGS = 'GRATINGS'¶

INSTRUMENTS = 'INSTRUMENTS'¶

LOOK_STATUS = 'LOOK_STATUS'¶

NUM_PER_PAGE = 'NUM_PER_PAGE'¶

PROPOSAL_CATEGORY = 'PROPOSAL_CATEGORY'¶

PUPILS = 'PUPILS'¶

READ_PATTS = 'READ_PATTS'¶

SORT_TYPE = 'SORT_TYPE'¶

SUBARRAYS = 'SUBARRAYS'¶

crds_tools.py¶

This module contains functions used to indentify and download reference files from CRDS and place them in the expected location, for JWQL to find.

This module uses the crds software package (https://hst-crds.stsci.edu/static/users_guide/index.html) which is installed when the JWST calibration pipeline package is installed. Reference files are identified by supplying some basic metadata from the exposure being calibrated. See https://hst-crds.stsci.edu/static/users_guide/library_use.html#crds-getreferences for a description of the function used for this task.

Author¶

Bryan Hilbert

Use¶

This module can be used as such:

from mirage.reference_files import crds
params = {'INSTRUME': 'NIRCAM', 'DETECTOR': 'NRCA1'}
reffiles = crds.get_reffiles(params)

jwql.utils.crds_tools.dict_from_yaml(yaml_dict)[source]¶

Create a dictionary to be used as input to the CRDS getreferences function from the nested dictionary created when a standard Mirage input yaml file is read in.

Parameters:

yaml_dictdict: Nested dictionary from reading in yaml file
Returns
——-
crds_dictdict: Dictionary of information necessary to select refernce files via getreferences().

jwql.utils.crds_tools.env_variables()[source]¶

Check the values of the CRDS-related environment variables

Returns:

crds_data_pathstr: Full path to the location of the CRDS reference files

jwql.utils.crds_tools.get_reffiles(parameter_dict, reffile_types, download=True)[source]¶

Determine CRDS’s best reference files to use for a particular observation, and download them if they are not already present in the CRDS_PATH. The determination is made based on the information in the parameter_dictionary.

Parameters:

parameter_dictdict: Dictionary of basic metadata from the file to be processed by the returned reference files (e.g. INSTRUME, DETECTOR, etc)
reffile_typeslist: List of reference file types to look up and download. These must be contained in CRDS’s list of reference file types.
downloadbool: If True (default), the identified best reference files will be downloaded. If False, the dictionary of best reference files will still be returned, but the files will not be downloaded. The use of False is primarily intended to support testing on Travis.

Returns:

reffile_mappingdict: Mapping of downloaded CRDS file locations

jwql.utils.crds_tools.path_check()[source]¶

Check that the CRDS_PATH environment variable is set. This will be the location to which CRDS reference files are downloaded. If the env variable is not set, default to use $HOME/crds_cache/

Returns:

crds_pathstr: Full path to the location of the CRDS reference files

jwql.utils.crds_tools.server_check()[source]¶: Check that the CRDS_SERVER_URL environment variable is set. This controls where Mirage will look for CRDS information. If the env variable is not set, set it to the JWST CRDS server.

credentials.py¶

Utility functions related to accessing remote services and databases.

Authors¶

Johannes Sahlmann

Lauren Chambers

Mees Fix

Use¶

This module can be imported as such:
import credentials
token = credentials.get_mast_token()

jwql.utils.credentials.get_mast_base_url(request=None)[source]¶

Return base url for mnemonic query.

Parameters:

requestHttpRequest object: Incoming request from the webpage

Returns:

tokenstr or None: Base url for MAST JWST EDB API.

jwql.utils.credentials.get_mast_token(request=None)[source]¶

Return MAST token from either Astroquery.Mast, webpage cookies, the JWQL configuration file, or an environment variable.

Parameters:

requestHttpRequest object: Incoming request from the webpage

Returns:

tokenstr or None: User-specific MAST token string, if available

instrument_properties.py¶

Collection of functions dealing with retrieving/calculating various instrument properties

Authors¶

Bryan Hilbert

Uses¶

This module can be imported and used as such:

from jwql.utils import instrument_properties as inst
amps = inst.amplifier_info('my_files.fits')

jwql.utils.instrument_properties.amplifier_info(filename, omit_reference_pixels=True)[source]¶

Calculate the number of amplifiers used to collect the data in a given file using the array size and exposure time of a single frame (This is needed because there is no header keyword specifying how many amps were used.)

Parameters:

filenamestr: Name of fits file to investigate
omit_reference_pixelsbool: If True, return the amp boundary coordinates excluding reference pixels

Returns:

num_ampsint: Number of amplifiers used to read out the data
amp_boundsdict: Dictionary of amplifier boundary coordinates. Keys are strings of the amp number (1-4). Each value is a list composed of two tuples. The first tuple gives the coordinates of the (minimum x, maximum x, and x step value), and the second tuple gives the (minimum y, maximum y, and y step value). These are set up such that a list of indexes for each amplifier can be generated by using np.mgrid[x_min: x_max: x_step, y_min: y_max: y_step]

jwql.utils.instrument_properties.calc_frame_time(instrument, aperture, xdim, ydim, amps, sample_time=1e-05)[source]¶

Calculate the readout time for a single frame of a given size and number of amplifiers. Note that for NIRISS and FGS, the fast readout direction is opposite to that in NIRCam, so we switch xdim and ydim so that we can keep a single equation.

Parameters:

instrumentstr: Name of the instrument being simulated
aperturestr: Name of aperture being simulated (e.g NRCA1_FULL). Currently this is only used to check for the FGS ACQ1 aperture, which uses a unique value of colpad below.
xdimint: Number of columns in the frame
ydimint: Number of rows in the frame
ampsint: Number of amplifiers used to read out the frame
sample_timefloat: Time to sample a pixel, in seconds. For NIRCam/NIRISS/FGS this is 10 microseconds = 1e-5 seconds

Returns:

frametimefloat: Readout time in seconds for the frame

jwql.utils.instrument_properties.get_obstime(filename)[source]¶

Extract the observation date and time from a fits file

Parameters:

filenamestr: Name of fits file

Returns:

obs_timedatetime.datetime: Observation date and time

jwql.utils.instrument_properties.mean_time(times)[source]¶

Given a list of datetime objects, calculate the mean time

Parameters ———_ times : list

List of datetime objects

Returns:

meantime ; datetime.datetime: Mean time of the input times

logging_functions.py¶

Logging functions for the jwql automation platform.

This module provides decorators to log the execution of modules. Log files are written to the logs/ directory in the jwql central storage area, named by module name and timestamp, e.g. monitor_filesystem/monitor_filesystem_2018-06-20-15:22:51.log

Authors¶

Catherine Martlin

Alex Viana (wfc3ql Version)

Matthew Bourque

Jason Neal

Use¶

To log the execution of a module, use:

import os
import logging

from jwql.logging.logging_functions import configure_logging
from jwql.logging.logging_functions import log_info
from jwql.logging.logging_functions import log_fail

@log_info
@log_fail
def my_main_function():
    pass

if __name__ == '__main__':

    module = os.path.basename(__file__).replace('.py', '')
    configure_logging(module)

    my_main_function()

Dependencies¶

The user must have a configuration file named config.json placed in the jwql directory and it must contain keys for log_dir and admin_account.

References¶

This code is adopted and updated from python routine logging_functions.py written by Alex Viana, 2013 for the WFC3 Quicklook automation platform.

jwql.utils.logging_functions.configure_logging(module)[source]¶

Configure the log file with a standard logging format.

Parameters:

modulestr: The name of the module being logged.
production_modebool: Whether or not the output should be written to the production environement.
pathstr: Where to write the log if user-supplied path; default to working dir.

Returns:

log_filestr: The path to the file where the log is written to.

jwql.utils.logging_functions.get_log_status(log_file)[source]¶

Returns the end status of the given log_file (i.e. SUCCESS or FAILURE)

Parameters:

log_filestr: The path to the file where the log is written to

Returns:

statusbool: The status of the execution of the script described by the log file (i.e. SUCCESS or FAILURE)

jwql.utils.logging_functions.log_fail(func)[source]¶

Decorator to log crashes in the decorated code.

Parameters:

funcfunc: The function to decorate.

Returns:

wrappedfunc: The wrapped function.

jwql.utils.logging_functions.log_info(func)[source]¶

Decorator to log useful system information.

This function can be used as a decorator to log user environment and system information. Future packages we want to track can be added or removed as necessary.

Parameters:

funcfunc: The function to decorate.

Returns:

wrappedfunc: The wrapped function.

jwql.utils.logging_functions.log_timing(func)[source]¶

Decorator to time a module or function within a code.

Parameters:

funcfunc: The function to time.

Returns:

wrappedfunc: The wrapped function. Will log the time.

jwql.utils.logging_functions.make_log_file(module)[source]¶

Create the log file name based on the module name.

The name of the log_file is a combination of the name of the module being logged and the current datetime.

Parameters:

modulestr: The name of the module being logged.
production_modebool: Whether or not the output should be written to the production environment.
pathstr: Where to write the log if user-supplied path; default to working dir.

Returns:

log_filestr: The full path to where the log file will be written to.

mast_utils.py¶

Various utility functions for interacting with MAST

Authors¶

Bryan Hilbert

Use¶

This module can be imported as such:

>>> import mast_utils
results = mast_utils.mast_query('nircam', 'NRCA1_FULL', 'NRC_DARK', 53005.1, 53005.2)

jwql.utils.mast_utils.instrument_inventory(instrument, dataproduct=['IMAGE', 'SPECTRUM', 'SED', 'TIMESERIES', 'VISIBILITY', 'EVENTLIST', 'CUBE', 'CATALOG', 'ENGINEERING', 'NULL'], add_filters=None, add_requests=None, caom=False, return_data=False)[source]¶

Get the counts for a given instrument and data product

Parameters:

instrument: str: The instrument name, i.e. one of [‘niriss’,’nircam’,’nirspec’, ‘miri’,’fgs’]
dataproduct: sequence, str: The type of data product to search
add_filters: dict: The (‘paramName’:’values’) pairs to include in the ‘filters’ argument of the request e.g. add_filters = {‘filter’:’GR150R’}
add_requests: dict: The (‘request’:’value’) pairs to include in the request e.g. add_requests = {‘pagesize’:1, ‘page’:1}
caom: bool: Query CAOM service
return_data: bool: Return the actual data instead of counts only

Returns:

int, dict: The number of database records that satisfy the search criteria or a dictionary of the data if return_data=True

jwql.utils.mast_utils.instrument_keywords(instrument, caom=False)[source]¶

Get the keywords for a given instrument service

Parameters:

instrument: str: The instrument name, i.e. one of [‘niriss’,’nircam’,’nirspec’, ‘miri’,’fgs’]
caom: bool: Query CAOM service

Returns:

pd.DataFrame: A DataFrame of the keywords

jwql.utils.mast_utils.jwst_inventory(instruments=['fgs', 'miri', 'nircam', 'niriss', 'nirspec'], dataproducts=['image', 'spectrum', 'cube'], caom=False, plot=False, output_dir=None)[source]¶

Gather a full inventory of all JWST data in each instrument service by instrument/dtype

Parameters:

instruments: sequence: The list of instruments to count
dataproducts: sequence: The types of dataproducts to count
caom: bool: Query CAOM service
plot: bool: Return a pie chart of the data
output_dir: str: Directory into which plots are saved

Returns:

astropy.table.table.Table: The table of record counts for each instrument and mode

jwql.utils.mast_utils.mast_query(instrument, templates, start_date, end_date, aperture=None, detector=None, filter_name=None, pupil=None, grating=None, readpattern=None, lamp=None)[source]¶

Use astroquery to search MAST for data for given observation templates over a given time range

Parameters:

instrumentstr: Instrument name (e.g. nircam)
templatesstr or list: Single, or list of, templates for the query (e.g. NRC_DARK, MIR_FLATMRS)
start_datefloat: Starting date for the search in MJD
end_datefloat: Ending date for the search in MJD
aperturestr: Detector aperture to search for (e.g. NRCA1_FULL)
detectorstr: Detector name (e.g. MIRIMAGE)
filter_namestr: Fitler element (e.g. F200W)
pupilstr: Pupil element (e.g. F323N)
gratingstr: Grating element (e.g. MIRROR)
readpatternstr: Detector read out pattern (e.g. NISRAPID)
lampstr: Lamp name (e.g. LINE2)

Returns:

query_resultslist: List of dictionaries containing the query results

jwql.utils.mast_utils.mast_query_miri(detector, aperture, templates, start_date, end_date)[source]¶

Use astroquery to search MAST for data for given observation templates over a given time range for MIRI. MIRI is different than the other instruments in that (to find full frame flats and darks at least) you need to use the detector name rather than the aperture name. There is no full frame aperture name for the MRS detectors.

Parameters:

detectorstr: Name of the detector to search for. One of MIRIMAGE, MIRIFULONG, MIRIFUSHORT.
aperturestr: Aperture name on the detector (e.g. MIRIM_FULL)
templatesstr or list: Single, or list of, templates for the query (e.g. NRC_DARK, MIR_FLATMRS)
start_datefloat: Starting date for the search in MJD
end_datefloat: Ending date for the search in MJD

Returns:

query_resultslist: List of dictionaries containing the query results

monitor_template.py¶

This module is intended to be a template to aid in creating new monitoring scripts and to demonstrate how to format them to fully utilize the jwql framework.

Each monitoring script must be executable from the command line (i.e. have a if '__name__' == '__main__' section), as well as have a “main” function that calls all other functions, methods, or modules (i.e. the entirety of the code is executed within the scope of the main function), as shown in this example.

Users may utilize the jwql framework functions for logging, setting permissions, parsing filenames, etc. (See related ``import``s).

Authors¶

Catherine Martlin

Matthew Bourque

Use¶

This module can be executed from the command line:
python monitor_template.py
Alternatively, it can be called from a python environment via the following import statements:
from monitor_template import main_monitor_function
from monitor_template import secondary_function

Dependencies¶

The user must have a configuration file named config.json placed in the jwql directory.

Notes¶

Any monitoring script written for jwql must adhere to the jwql style guide located at: https://github.com/spacetelescope/jwql/blob/main/style_guide/README.md

jwql.utils.monitor_template.monitor_template_main()[source]¶: The main function of the monitor_template module.

jwql.utils.monitor_template.second_function(input_value)[source]¶

This is your axiliary function; you may have many of these.

Parameters:

input_valuestr: Some value to modify in the function.

Returns:

useful_resultstr: The result of modifying the input value.

monitor_utils.py¶

Various utility functions for instrument monitors

Authors¶

Matthew Bourque

Bryan Hilbert

Maria Pena-Guerrero

Use¶

This module can be imported as such:

>>> import monitor_utils
settings = monitor_utils.update_monitor_table('dark_monitor')

jwql.utils.monitor_utils.exclude_asic_tuning(mast_results)[source]¶

Given a list of file information from a MAST query, filter out files taken during ASIC tuning, which will have bad data in terms of results for the instrument monitors.

Parameters:

mast_resultslist: List of dictionaries containing a MAST query result

Returns:

filtered_resultslist: Modified list with ASIC tuning entries removed

jwql.utils.monitor_utils.initialize_instrument_monitor(module)[source]¶

Configures a log file for the instrument monitor run and captures the start time of the monitor

Parameters:

modulestr: The module name (e.g. dark_monitor)

Returns:

start_timedatetime object: The start time of the monitor
log_filestr: The path to where the log file is stored

jwql.utils.monitor_utils.mast_query_darks(instrument, aperture, start_date, end_date, readpatt=None)[source]¶

Use astroquery to search MAST for dark current data

Parameters:

instrumentstr: Instrument name (e.g. nircam)
aperturestr: Detector aperture to search for (e.g. NRCA1_FULL)
start_datefloat: Starting date for the search in MJD
end_datefloat: Ending date for the search in MJD
readpattstr: Readout pattern to search for (e.g. RAPID). If None, readout pattern will not be added to the query parameters.

Returns:

query_resultslist: List of dictionaries containing the query results

jwql.utils.monitor_utils.mast_query_ta(instrument, aperture, start_date, end_date, readpatt=None)[source]¶

Use astroquery to search MAST for TA current data

Parameters:

instrumentstr: Instrument name (e.g. nirspec)
aperturestr: Detector aperture to search for (e.g. NRS_S1600A1_SLIT)
start_datefloat: Starting date for the search in MJD
end_datefloat: Ending date for the search in MJD
readpattstr: Readout pattern to search for (e.g. RAPID). If None, readout pattern will not be added to the query parameters.

Returns:

query_resultslist: List of dictionaries containing the query results

jwql.utils.monitor_utils.model_query_ta(instrument, aperture, start_date, end_date, readpatt=None)[source]¶

Use local Django model to search for TA data.

Parameters:

instrumentstr: Instrument name (e.g. nirspec)
aperturestr: Detector aperture to search for (e.g. NRS_S1600A1_SLIT)
start_datefloat: Starting date for the search in MJD
end_datefloat: Ending date for the search in MJD
readpattstr: Readout pattern to search for (e.g. RAPID). If None, readout pattern will not be added to the query parameters.

Returns:

query_resultslist: List of dictionaries containing the query results

jwql.utils.monitor_utils.update_monitor_table(module, start_time, log_file)[source]¶

Update the monitor database table with information about the instrument monitor run

Parameters:

modulestr: The module name (e.g. dark_monitor)
start_timedatetime object: The start time of the monitor
log_filestr: The path to where the log file is stored

organize_filesystem.py¶

This module takes a collection of JWST FITS files and moves the files into a MAST-data-cache-like filesystem.

The files that the module will process is provided by the old_filesysem key in the config.json file. The files will be moved into the directory provided by the filesystem key in the config.json file.

For example, the file <old_filesystem>/jw00312/jw00312002001_02102_00001_nrcb4_rateints.fits

will be placed in the directory <filesystem>/jw00312/jw00312002001/.

Authors¶

Matthew Bourque

Use¶

This module is intended to be executed via the command line as such:
python organize_filesystem.py

jwql.utils.organize_filesystem.organize_filesystem()[source]¶: The main function of the organize_filesystem module. See module docstrings for further details.

jwql.utils.organize_filesystem.revert_filesystem()[source]¶

Perform the opposite of organize_filesystem – this function will move files from a MAST-data-cache-like organization to the previous organization.

For example, the file <filesystem>/jw00312/jw00312002001/jw00312002001_02102_00001_nrcb4_rateints.fits` will be placed in the directory ``<old_filesystem>/jw00312/.

permissions.py¶

Permissions module for managing file permissions for jwql.

This module provides jwql with functions to inspect and set file permissions.

The module takes as input a path to a file or directory, checks whether the owner of the file is the jwql admin account, and if so, (1) set the permissions appropriately, and (2) set the group membership appropriately.

Authors¶

Johannes Sahlmann

Use¶

This module can be imported and used with
from jwql.permissions import permissions
permissions.set_permissions(pathname)
Required arguments:

pathname - Directory or file for which the default permissions should be set

Notes¶

Permissions are set and read using the stat module, see https://docs.python.org/3/library/stat.html

Below is a list with the relevant stat attribute names, integer, octal, and string representations.

stat_key stat_mode stat_mode_octal stat_mode_string
-------- --------- --------------- ----------------
S_IFPORT         0             0o0       ?---------
S_IFDOOR         0             0o0       ?---------
 S_IXOTH         1             0o1       ?--------x
 S_IWOTH         2             0o2       ?-------w-
 S_IROTH         4             0o4       ?------r--
 S_IRWXO         7             0o7       ?------rwx
 S_IXGRP         8            0o10       ?-----x---
 S_IWGRP        16            0o20       ?----w----
 S_IRGRP        32            0o40       ?---r-----
 S_IRWXG        56            0o70       ?---rwx---
 S_IEXEC        64           0o100       ?--x------
 S_IXUSR        64           0o100       ?--x------
 S_IWUSR       128           0o200       ?-w-------
S_IWRITE       128           0o200       ?-w-------
 S_IREAD       256           0o400       ?r--------
 S_IRUSR       256           0o400       ?r--------
 S_IRWXU       448           0o700       ?rwx------
 S_ISVTX       512          0o1000       ?--------T
 S_ISGID      1024          0o2000       ?-----S---
 S_ISUID      2048          0o4000       ?--S------
 S_IFIFO      4096             0o0       p---------
 S_IFCHR      8192             0o0       c---------
 S_IFDIR     16384             0o0       d---------
 S_IFBLK     24576             0o0       b---------
 S_IFREG     32768             0o0       ----------
 S_IFLNK     40960             0o0       l---------
S_IFSOCK     49152             0o0       s---------
 S_IFWHT     57344             0o0       w---------

jwql.utils.permissions.find_mode_to_use(pathname, owner, mode)[source]¶

Select the appropriate mode to use for the input pathname, depending on who the owner is, as well as whether the pathname is a file or directory.

Parameters:

pathnamestr: Directory or file to be inspected
ownerstr: String representation of the owner
modedict: Dictionary of integer representation of the permission mode, compatible with os.stat output. Keys are <owner name> (with a default of DEFAULT_OWNER), and ‘other’, so we can differentiate between files created by the server accounts and those created by local users

Returns:

mode_valueint: Integer representation of the permission mode, compatible with os.stat output

jwql.utils.permissions.get_group_string(pathname)[source]¶

Return the group of pathname in string representation.

Parameters:

pathnamestr: Directory or file to be inspected.

Returns:

group_namestr: String representation of the group.

jwql.utils.permissions.get_owner_string(pathname)[source]¶

Return the owner of pathname in string representation.

Parameters:

pathnamestr: Directory or file to be inspected

Returns:

owner_namestr: String representation of the owner.

jwql.utils.permissions.has_permissions(pathname, owner='jwqladm', mode={'jwqladm': 420, 'other': 436}, group='jwql_dev')[source]¶

Return boolean indicating whether pathname has the specified owner, permission, and group scheme.

Parameters:

pathnamestr: Directory or file to be inspected
ownerstr: String representation of the owner
modeint: Integer representation of the permission mode, compatible with os.stat output
groupstr: String representation of the group

Returns:

booleanbool

jwql.utils.permissions.set_permissions(pathname, owner='jwqladm', mode={'jwqladm': 420, 'other': 436}, group='jwql_dev', verbose=False)[source]¶

Set mode and group of the file/directory identfied by pathname, if and only if it is owned by owner.

Parameters:

pathnamestr: Directory or file to be inspected
ownerstr: String representation of the owner
modedict: Dictionary of integer representation of the permission mode, compatible with os.stat output. Keys are <owner name> (with a default of DEFAULT_OWNER), and ‘other’, so we can differentiate between files created by the server accounts and those created by local users
groupstr: String representation of the group
verbosebool: Boolean indicating whether verbose output is requested

jwql.utils.permissions.show_permissions(pathname)[source]¶

Verbose output showing group, user, and permission information for a directory or file.

Parameters:

pathnamestr: Directory or file to be inspected

jwql.utils.permissions.verify_path(pathname)[source]¶

Verify that pathname is either a directory or a file. If not, an error is raised.

Parameters:

pathnamestr: Directory or file to be inspected

plotting.py¶

This module is a collection of plotting functions that may be used across the jwql application.

Authors:¶

Joe Filippazzo

Use:¶

This module can be use as follows:

from jwql.utils import plotting
from pandas import DataFrame
data = DataFrame({'meow': {'foo': 12, 'bar': 23, 'baz': 2},
                  'mix': {'foo': 45, 'bar': 31, 'baz': 23},
                  'deliver': {'foo': 62, 'bar': 20, 'baz': 9}})
data = data.reset_index()
plt = plotting.bar_chart(data, 'index')

jwql.utils.plotting.bar_chart(dataframe, groupcol, datacols=None, **kwargs)[source]¶

Create a pie chart from a Pandas DataFrame

Parameters:

dataframepandas.DataFrame: A dataframe of values
groupcolstr: The name of the column with the group labels
datacolstr, sequence (optional): The name or list of names of the column containing the data. In None, uses all columns except groupcol

Returns:

pltobj: The generated bokeh.figure object

preview_image.py¶

Create a preview image from a fits file containing an observation.

This module creates and saves a “preview image” from a fits file that contains a JWST observation. Data from the user-supplied extension of the file are read in, along with the PIXELDQ extension if present. For each integration in the exposure, the first group is subtracted from the final group in order to create a difference image. The lower and upper limits to be displayed are defined as the clip_percent and (1. - clip_percent) percentile signals. matplotlib is then used to display a linear- or log-stretched version of the image, with accompanying colorbar. The image is then saved.

Authors:¶

Bryan Hilbert

Use:¶

This module can be imported as such:

from jwql.preview_image.preview_image import PreviewImage
im = PreviewImage(my_file, "SCI")
im.clip_percent = 0.01
im.scaling = 'log'
im.output_format = 'jpg'
im.make_image()

class jwql.utils.preview_image.PreviewImage(filename, extension)[source]¶

An object for generating and saving preview images, used by generate_preview_images.

Attributes:

clip_percentfloat: The amount to sigma clip the input data by when scaling the preview image. Default is 0.01.
cmapstr: The colormap used by matplotlib in the preview image. Default value is viridis.
dataobj: The data used to generate the preview image.
dqobj: The DQ data used to generate the preview image.
filestr: The filename to generate the preview image from.
output_formatstr: The format to which the preview image is saved. Options are jpg and thumb
preview_output_directorystr or None: The output directory to which the preview image is saved.
scalingstr: The scaling used in the preview image. Default is log.
thumbnail_output_directorystr or None: The output directory to which the thumbnail is saved.

Methods

difference_image(data)	Create a difference image from the data
find_limits(data, pixmap, clipperc)	Find the min and max signal levels after clipping by `clipperc`
get_data(filename, ext)	Read in data from the given `filename` and `ext`
make_figure(image, integration_number, min_value, max_value, scale, maxsize, thumbnail)	Create the `matplotlib` figure
make_image(max_img_size)	Main function
save_image(fname, thumbnail)	Save the figure

determine_map_file(header)[source]¶

Determine which file contains the map of non-science pixels given a file header

Parameters:

headerastropy.io.fits.header: Header object from an HDU object

difference_image(data)[source]¶

Create a difference image from the data. Use last group minus first group in order to maximize signal to noise. With 4D input, make a separate difference image for each integration.

Parameters:

dataobj: 4D numpy ndarray array of floats

Returns:

resultobj: 3D numpy ndarray containing the difference image(s) from the input exposure

find_limits(data)[source]¶

Find the minimum and maximum signal levels after clipping the top and bottom clipperc of the pixels.

Parameters:

dataobj: 2D numpy ndarray of floats

Returns:

resultstuple: Tuple of floats, minimum and maximum signal levels

get_data(filename, ext)[source]¶

Read in the data from the given file and extension. Also find how many rows/cols of reference pixels are present.

Parameters:

filenamestr: Name of fits file containing data
extstr: Extension name to be read in

Returns:

dataobj: Science data from file. A 2-, 3-, or 4D numpy ndarray
dqobj: 2D ndarray boolean map of reference pixels. Science pixels flagged as True and non-science pixels are False

get_nonsci_map(hdulist, extensions, xdim, ydim)[source]¶

Create a map of non-science pixels for a given HDUList. If there is no DQ extension in the HDUList, assume all pixels are science pixels.

Parameters:

hdulistastropy.io.fits.HDUList: HDUList object from a fits file
extensionslist: List of extension names in the HDUList
xdimint: Number of columns in data array. Only used if there is no DQ extension
ydimint: Number of rows in the data array. Only used if there is no DQ extension

Returns:

dqnumpy.ndarray: 2D boolean array giving locations of non-science pixels

make_figure(image, integration_number, min_value, max_value, scale, maxsize=8, thumbnail=False)[source]¶

Create the matplotlib figure of the image

Parameters:

imageobj: 2D numpy ndarray of floats
integration_numberint: Integration number within exposure
min_valuefloat: Minimum value for display
max_valuefloat: Maximum value for display
scalestr: Image scaling (log, linear)
maxsizeint: Size of the longest dimension of the output figure (inches)
thumbnailbool: True to create a thumbnail image, False to create the full preview image

Returns:

resultobj: Matplotlib Figure object

make_image(max_img_size=8.0, create_thumbnail=False)[source]¶

The main function of the PreviewImage class.

Parameters:

max_img_sizefloat: Image size in the largest dimension
create_thumbnailbool: If True, a thumbnail image is created and saved.

nonsci_from_file()[source]¶

Read in a map of non-science/reference pixels from a fits file

Parameters:

filenamestr: Name of fits file to be read in.

Returns:

mapnumpy.ndarray: 2D boolean array of pixel values

save_image(fname, thumbnail=False)[source]¶

Save an image in the requested output format and sets the appropriate permissions

Parameters:

imageobj: A matplotlib figure object
fnamestr: Output filename
thumbnailbool: True if saving a thumbnail image, false for the full preview image.

jwql.utils.preview_image.create_nir_nonsci_map()[source]¶

Create a map of non-science pixels for a near-IR detector

Returns:

arrnumpy.ndarray: 2D boolean array. Science pixels have a value of 1 and non-science pixels (reference pixels) have a value of 0.

jwql.utils.preview_image.crop_to_subarray(arr, header, xdim, ydim)[source]¶

Given a full frame array, along with a fits HDU header containing subarray information, crop the array down to the indicated subarray.

Parameters:

arrnumpy.ndarray: 2D array of data. Assumed to be full frame (2048 x 2048)
headerastropy.io.fits.header: Header from a single extension of a fits file
xdimint: Number of columns in the corresponding data (not dq) array, in pixels
ydimint: Number of rows in the corresponding data (not dq) array, in pixels

Returns:

arrnumpy.ndarray: arr, cropped down to the size specified in the header

jwql.utils.preview_image.expand_for_i2d(array, xdim, ydim)[source]¶

Some file types, like i2d files, contain arrays with sizes that are different than those specified in the SUBSIZE header keywords. In those cases, we need to expand the input array from the official size to the actual size.

Parameters:

arraynumpy.ndarray: 2D DQ array of booleans
xdimint: Number of columns in the data whose dimensions we want array to have. (e.g. the dimensions of the i2d file)
ydimint: Number of rows in the data whose dimensions we want array to have. (e.g. the dimensions of the i2d file)

Returns:

new_arraynumpy.ndarray: 2D array with dimensions of (ydim x xdim)

jwql.utils.preview_image.nan_to_zero(image)[source]¶

Set any pixels with a value of NaN to zero

Parameters:

imagenumpy.ndarray: Array from which NaNs will be removed

Returns:

imagenumpy.ndarray: Input array with NaNs changed to zero

utils.py¶

Various utility functions for the jwql project.

Authors¶

Matthew Bourque

Lauren Chambers

Use¶

This module can be imported as such:
>>> import utils
settings = get_config()

References¶

Filename parser modified from Joe Hunkeler: https://gist.github.com/jhunkeler/f08783ca2da7bfd1f8e9ee1d207da5ff

Various documentation related to JWST filename conventions: - https://innerspace.stsci.edu/pages/viewpage.action?pageId=94092600 - https://innerspace.stsci.edu/pages/viewpage.action?spaceKey=SCSB&title=JWST+Science+Data+Products - https://jwst-pipeline.readthedocs.io/en/stable/jwst/introduction.html#pipeline-step-suffix-definitions - JWST TR JWST-STScI-004800, SM-12

jwql.utils.utils.check_config_for_key(key)[source]¶

Check that the config.json file contains the specified key and that the entry is not empty

Parameters:

keystr: The configuration file key to verify

jwql.utils.utils.copy_files(files, out_dir)[source]¶

Copy a given file to a given directory. Only try to copy the file if it is not already present in the output directory.

Parameters:

fileslist: List of files to be copied
out_dirstr: Destination directory

Returns:

successlist: Files successfully copied (or that already existed in out_dir)
failedlist: Files that were not copied

jwql.utils.utils.create_png_from_fits(filename, outdir)[source]¶

Create and save a png file of the provided file. The file will be saved with the same filename as the input file, but with fits replaced by png

Parameters:

filenamestr: Fits file to be opened and saved as a png
outdirstr: Output directory to save the png file to

Returns:

png_filestr: Name of the saved png file

jwql.utils.utils.delete_non_rate_thumbnails(extensions=['_rate_', '_dark'])[source]¶

This script will go through all the thumbnail directories and delete all thumbnails that do not contain the given extensions. We currently create thumbnails using only rate.fits and dark.fits files, so the default is to keep only those.

Parameters:

extensionlist: If a thumbnail filename contains any of these strings, it will not be deleted

jwql.utils.utils.download_mast_data(query_results, output_dir)[source]¶

Example function for downloading MAST query results. From MAST website (https://mast.stsci.edu/api/v0/pyex.html)

Parameters:

query_resultslist: List of dictionaries returned by a MAST query.
output_dirstr: Directory into which the files will be downlaoded

jwql.utils.utils.ensure_dir_exists(fullpath)[source]¶: Creates dirs from fullpath if they do not already exist.

jwql.utils.utils.filename_parser(filename)[source]¶

Return a dictionary that contains the properties of a given JWST file (e.g. program ID, visit number, detector, etc.).

Parameters:

filenamestr: Path or name of JWST file to parse

Returns:

filename_dictdict: Collection of file properties

Raises:

ValueError: When the provided file does not follow naming conventions

jwql.utils.utils.filesystem_path(filename, check_existence=True, search=None)[source]¶

Return the path to a given file in the filesystem.

The full path is returned if check_existence is True, otherwise only the path relative to the filesystem key in the config.json file is returned.

Parameters:

filenamestr: File to locate (e.g. jw86600006001_02101_00008_guider1_cal.fits)
check_existenceboolean: Check to see if the file exists in the expected lcoation
searchstr: A search term to use in a glob.glob statement if the full filename is unkown (e.g. *rate.fits). In this case, the first element of the list of returned values is chosen as the filename.

Returns:

full_pathstr: Full path to the given file, including filename

jwql.utils.utils.get_base_url()[source]¶

Return the beginning part of the URL to the jwql web app based on which user is running the software.

If the admin account is running the code, the base_url is assumed to be the production URL. If not, the base_url is assumed to be local.

Returns:

base_urlstr: The beginning part of the URL to the jwql web app

jwql.utils.utils.get_config()[source]¶

Return a dictionary that holds the contents of the jwql config file.

Returns:

settingsdict: A dictionary that holds the contents of the config file.

jwql.utils.utils.get_rootnames_for_instrument_proposal(instrument, proposal)[source]¶

Return a list of rootnames for the given instrument and proposal

Parameters:

instrumentstr: Name of the JWST instrument, with first letter capitalized (e.g. Fgs)
proposalint or str: Proposal ID number

Returns:

rootnameslist: List of rootnames for the given instrument and proposal number

jwql.utils.utils.grouper(iterable, chunksize)[source]¶

Take a list of items (iterable), and group it into chunks of chunksize, with the last chunk being any remaining items. This allows you to batch-iterate through a potentially very long list without missing any items, and where each individual iteration can involve a much smaller number of files. Particularly useful for operations that you want to execute in batches, but don’t want the batches to be too long.

Examples

grouper([1, 2, 3, 4, 5], 2) produces (1, 2), (3, 4), (5, )

grouper([1, 2, 3, 4, 5], 6) produces (1, 2, 3, 4, 5)

jwql.utils.utils.query_format(string)[source]¶: Take a string of format lower_case and change it to UPPER CASE

jwql.utils.utils.query_unformat(string)[source]¶: Take a string of format UPPER CASE and change it to UPPER_CASE

jwql.utils.utils.read_png(filename)[source]¶

Open the given png file and return as a 3D numpy array

Parameters:

filenamestr: png file to be opened

Returns:

datanumpy.ndarray: 3D array representation of the data in the png file

jwql.utils.utils.save_png(fig, filename='')[source]¶

Starting with selenium version 4.10.0, our testing has shown that on the JWQL servers, we need to specify an instance of a web driver when exporting a Bokeh figure as a png. This is a wrapper function that creates the web driver instance and calls Bokeh’s export_png function.

Parameters:

figbokeh.plotting.figure: Bokeh figure to be saved as a png
filenamestr: Filename to use for the png file