common_monitors

bad_pixel_monitor.py

This module contains code for the bad/dead pixel monitor.

The monitor calls the bad_pixel_mask.py module in the spacetelescope/jwst_reffiles package in order to identify bad pixels. (https://github.com/spacetelescope/jwst_reffiles)

The bad pixel mnemonics to be searched for are given in the jwst package: https://jwst-pipeline.readthedocs.io/en/stable/jwst/references_general/ references_general.html#data-quality-flags

The algorithms used to identify the various types of bad pixels were developed by the JWST Calibration Reference File Generation Tools working group, which was under the Calibration Working Group. This working group contained representatives from all instruments, and worked to identify common algorithms that all instruments could use to generate information to be placed in reference files. Details on the algorithms used here to identify bad pixels are detailed here: https://outerspace.stsci.edu/display/JWSTCC/Algorithm+details%3A+DQ+Init

The bad pixel search is composed of two different parts, each of which can be run independently.

1. Internal flat field exposures are used to search for DEAD, LOW QE, OPEN, and ADJACENT TO OPEN pixels.

2. Dark current exposures are used to search for NOISY, HOT, RC, TELEGRAPH, and LOW_PEDESTAL pixels.

Both of these modules expect input data in at least 2 calibration states. In practice, the bad pixel monitor will search MAST for the appropriate dark current and flat field files. In both cases, a given file is considered useful if the uncal (and potentially the rate) versions of the file are present in the archive. For files where the uncal version only is found, the pipeline is run to produce the rate file.

Once a sufficient number of new flats or darks are identified, the bad pixel montior is called. The bad_pixel_file_thresholds.txt file contains a list of the minimum number of new files necessary to run the monitor for each instrument and aperture.

For the flat field files, the pipeline is run on the uncal files far enough to produce cosmic ray flagged (jump) files. These are also needed for the bad pixel search.

The jwst_reffiles bad_pixel_mask.py is run, and returns a map of bad pixels for each of the various bad pixel mnemonics. The bad_pixel_monitor then downloads the latest bad pixel mask in CRDS for the given instrument and detector/aperture, and this is compared to the new map of bad pixels. For each bad pixel mnemonic, any pixels flagged as bad that are not bad in the current reference file are saved to the (e.g. NIRCamBadPixelStats) database table.

Author

  • Bryan Hilbert

Use

This module can be used from the command line as such:

python bad_pixel_monitor.py

Notes

The bad pixel flat that utilizes flat field ramps can’t be used with NIRCam since NIRCam has no internal lamp and therefore will not be taking any more internal flat field images. Could perhaps be used with a series of external undithered observations, but that’s something to think about later.

Templates to use: FGS_INTFLAT, NIS_LAMP, NRS_LAMP, MIR_DARK

class jwql.instrument_monitors.common_monitors.bad_pixel_monitor.BadPixels[source]

Class for executing the bad pixel monitor.

This class will search for new (since the previous instance of the class) dark current and internal flat field files in the filesystem. It will loop over instrument/aperture combinations and find the number of new dark/flat files available. If there are enough, it will copy the files over to a working directory and run the monitor.

This will use the jwst_reffiles package to locate new bad pixels, which will be returned as a map. This map will be compared to the current bad pixel reference file (dq_init) in CRDS, and any the coordinates and type of any new bad pixels will be saved in a database table.

Raises:
ValueError

If NINT or DETECTOR is missing from input file header

ValueError

If an unrecognized bad pixel mnemonic is encountered

ValueError

If the number of uncal and rate files does not match

ValueError

If the most recent query search returns more than one entry

Attributes:
aperturestr

Aperture name of the data (e.g. NRCA1_FULL)

dark_query_startfloat

Date (in MJD) of the ending range of the previous MAST query where the bad pixel from darks monitor was run.

data_dirstr

Directory that contains the files copied from MAST to be used by the bad pixel monitor

detectorstr

Detector associated with the data (e.g. NRCA1)

flat_query_startfloat

Date (in MJD) of the ending range of the previous MAST query where the bad pixel from flats monitor was run.

instrumentstr

Name of the JWST instrument the data are from

nintsint

Number of integrations in the exposure

output_dirstr

Top level output directory associated with the bad pixel monitor, as retrieved from the JWQL config file

pixel_tablesqlalchemy table

Database table containing lists of bad pixels identified during runs of the bad pixel monitor

query_endfloat

MJD of the execution time of the bad pixel monitor. This is used as the ending time of the MAST query.

query_tablesqlalchemy table

Database table containing the history of MAST queries for the bad pixel monitor.

add_bad_pix(coordinates, pixel_type, files, obs_start_time, obs_mid_time, obs_end_time, baseline_file)[source]

Add a set of bad pixels to the bad pixel database table

Parameters:
coordinatestuple

Tuple of two lists, containing x,y coordinates of bad pixels (Output of np.where call)

pixel_typestr

Type of bad pixel. e.g. dead, hot, and noisy

fileslist

List of fits files which were used to identify the bad pixels

obs_start_timedatetime.datetime

Observation time of the earliest file in files

obs_mid_timedatetime.datetime

Average of the observation times in files

obs_end_timedatetime.datetime

Observation time of the latest file in files

baseline_filestr

Name of baseline bad pixel file against which the new bad pixel population was compared

exclude_existing_badpix(badpix, pixel_type)[source]

Given a set of coordinates of bad pixels, determine which of these pixels have been previously identified and remove them from the list

Parameters:
badpixtuple

Tuple of lists containing x and y pixel coordinates. (Output of numpy.where call)

pixel_typestr

Type of bad pixel being examined. Options are hot, dead, and noisy

Returns:
new_pixels_xlist

List of x coordinates of new bad pixels

new_pixels_ylist

List of y coordinates of new bad pixels

filter_query_results(results, datatype)[source]

Filter MAST query results. For input flats, keep only those with the most common filter/pupil/grating combination. For both flats and darks, keep only those with the most common readout pattern.

Parameters:
resultslist

List of query results, as returned by mast_query()

datatypestr

Type of data being filtered. flat or dark.

Returns:
readpatt_filteredlist

Filtered list of query results.

get_metadata(filename)[source]

Collect basic metadata from a fits file

Parameters:
filenamestr

Name of fits file to examine

get_possible_apertures()[source]

Generate a list of possible SIAF apertures for the given instrument.

Returns:
possible_aperturelist

List of acceptible apertures for self.instrument

identify_tables()[source]

Determine which database tables to use for a run of the bad pixel monitor

make_crds_parameter_dict()[source]

Construct a paramter dictionary to be used for querying CRDS

Returns:
parametersdict

Dictionary of parameters, in the format expected by CRDS

map_uncal_and_rate_file_lists(uncal_files, rate_files, rate_files_to_copy, obs_type)[source]

Copy uncal and rate files from the filesystem to the working directory. Any requested files that are not in the filesystem are noted and skipped. Return the file lists with skipped files removed.

Parameters:
uncal_fileslist

List of raw files to be copied

rate_fileslist

List of rate (slope) images to be copied. This list should correspond 1-to-1 with uncal_files. Any rate files that were not found in the MAST query should be set to None.

rate_files_to_copylist

Similar to rate_files but with the None entries omitted.

obs_typestr

Observation type (dark or flat). Used only for logging

Returns:
uncal_fileslist

List of the input raw files with any that failed to copy removed

rate_fileslist

List of the input rate files with any that failed to copy removed (if the uncal also failed) or set to None (if only the rate file failed)

Query the query history database and return the information on the most recent query for the given aperture_name where the dark monitor was executed.

Parameters:
file_typestr

dark or flat. Specifies the type of file whose previous search time is queried.

Returns:
query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the dark monitor was run.

process(illuminated_raw_files, illuminated_slope_files, dark_raw_files, dark_slope_files)[source]

The main method for processing darks. See module docstrings for further details.

Parameters:
illuminated_raw_fileslist

List of filenames (including full paths) of raw (uncal) flat field files. These should all be for the same detector and aperture.

illuminated_slope_fileslist

List of filenames (including full paths) of flat field slope files. These should all be for the same detector and aperture and correspond one-to-one with illuminated_raw_files. For cases where a raw file exists but no slope file, the slope file should be None

dark_raw_fileslist

List of filenames (including full paths) of raw (uncal) dark files. These should all be for the same detector and aperture.

dark_slope_fileslist

List of filenames (including full paths) of dark current slope files. These should all be for the same detector and aperture and correspond one-to-one with dark_raw_files. For cases where a raw file exists but no slope file, the slope file should be None

run()[source]

The main method. See module docstrings for further details.

There are 2 parts to the bad pixel monitor: 1. Bad pixels from illuminated data 2. Bad pixels from dark data

For each, we will query MAST, copy new files from the filesystem and pass the list of copied files into the process() method.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.bad_map_to_list(badpix_image, mnemonic)[source]

Given an DQ image and a bad pixel mnemonic, create a list of (x,y) locations of this type of bad pixel in badpix_image

Parameters:
badpix_imagenumpy.ndarray

2D image of bad pixels (i.e. a DQ array)

mnemonicstr

The type of bad pixel to map. The mnemonic must be one of those in the JWST calibration pipeline’s list of possible mnemonics

Returns:
x_loclist

List of x locations within badpix_image containing mnemonic pixels.

y_loclist

List of x locations within badpix_image containing mnemonic pixels.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.check_for_sufficient_files(uncal_files, instrument_name, aperture_name, threshold_value, file_type)[source]

From a list of files of a given type (flats or darks), check to see if there are enough files to call the bad pixel monitor. The number of files must be equal to or greater than the provided threshold_value.

Parameters:
uncal_fileslist

List of filenames

instrument_namestr

Name of JWST instrument (e.g. nircam) that the data are from. This is used only in logging statements

aperture_namestr

Name of aperture (e.g.`` NRCA1_FULL``) that the data are from. This is used only in logging statements

threshold_valueint

Minimum number of files required in order to run the bad pixel monitor

file_typestr

Either darks` or flats. This is used only in the logging statements.

Returns:
uncal_fileslist

List of sorted, unique files from the input file list. Set to None if the number of unique files is under the threshold

run_databool

Whether or not the bad pixel monitor will be called on these files.

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.exclude_crds_mask_pix(bad_pix, existing_bad_pix)[source]

Find differences between a set of newly-identified bad pixels and an existing set. Return a list of newly-discovered bad pixels that are not present in the existing set.

Parameters:
bad_pixnumpy.ndarray

2D array of bad pixel flags. Flags must correspond to the defintiions used by the JWST calibration pipeline

existing_bad_pixnumpy.ndarray

2D array of bad pixel flags. Flags must correspond to the definitions used by the JWST calibration pipeline

Returns:
new_bad_pixnumpy.ndarray

2D array of bad pixel flags contained in bad_pix but not existing_bad_pix

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.locate_rate_files(uncal_files)[source]

Given a list of uncal (raw) files, generate a list of corresponding rate files. For each uncal file, if the rate file is present in the filesystem, add the name of the rate file (if a rateints file exists, use that) to the list of files. If no rate file is present, add None to the list.

Parameters:
uncal_fileslist

List of uncal files to use as the basis of the search

Returns:
rate_fileslist

List of rate files. This list corresponds 1-to-1 with uncal_files. Any missing rate files are listed as None.

rate_files_to_copylist

Same as rate_files but without the None entries. This is a list of only the rate files that exist in the filesystem

jwql.instrument_monitors.common_monitors.bad_pixel_monitor.locate_uncal_files(query_result)[source]

Given a MAST query result, locate the raw version (uncal.fits) of the listed files in the filesystem.

Parameters:
query_resultlist

MAST query results. List of dictionaries

Returns:
uncal_fileslist

List of raw file locations within the filesystem

bias_monitor.py

This module contains code for the bias monitor, which monitors the bias levels in dark exposures as well as the performance of the pipeline superbias subtraction over time.

For each instrument, the 0th group of full-frame dark exposures is saved to a fits file. The median signal levels in these images are recorded in the <Instrument>BiasStats database table for the odd/even rows/columns of each amp.

Next, these images are run through the jwst pipeline up through the reference pixel correction step. These calibrated images are saved to a fits file as well as a png file for visual inspection of the quality of the pipeline calibration. A histogram distribution of these images, as well as their collapsed row/column and sigma-clipped mean and standard deviation values, are recorded in the <Instrument>BiasStats database table.

Author

  • Ben Sunnquist

  • Maria A. Pena-Guerrero

Use

This module can be used from the command line as such:

python bias_monitor.py
class jwql.instrument_monitors.common_monitors.bias_monitor.Bias[source]

Class for executing the bias monitor.

This class will search for new full-frame dark current files in the file system for each instrument and will run the monitor on these files. The monitor will extract the 0th group from the new dark files and output the contents into a new file located in a working directory. It will then perform statistical measurements on these files before and after pipeline calibration in order to monitor the bias levels over time as well as ensure the pipeline superbias is sufficiently calibrating new data. Results are all saved to database tables.

Attributes:
output_dirstr

Path into which outputs will be placed.

data_dirstr

Path into which new dark files will be copied to be worked on.

query_startfloat

MJD start date to use for querying MAST.

query_endfloat

MJD end date to use for querying MAST.

instrumentstr

Name of instrument used to collect the dark current data.

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL).

collapse_image(image)[source]

Median-collapse the rows and columns of an image.

Parameters:
imagenumpy.ndarray

2D array on which to calculate statistics.

Returns:
collapsed_rowsnumpy.ndarray

1D array of the collapsed row values.

collapsed_columnsnumpy.ndarray

1D array of the collapsed column values.

determine_pipeline_steps()[source]

Determines the necessary JWST pipelines steps to run on a given dark file.

Returns:
pipeline_stepscollections.OrderedDict

The pipeline steps to run.

extract_zeroth_group(filename)[source]

Extracts the 0th group of a fits image and outputs it into a new fits file.

Parameters:
filenamestr

The fits file from which the 0th group will be extracted.

Returns:
output_filenamestr

The full path to the output file.

file_exists_in_database(filename)[source]

Checks if an entry for filename exists in the bias stats database.

Parameters:
filenamestr

The full path to the uncal filename.

Returns:
file_existsbool

True if filename exists in the bias stats database.

get_amp_medians(image, amps)[source]

Calculates the median in the input image for each amplifier and for odd and even rows/columns separately.

Parameters:
imagenumpy.ndarray

2D array on which to calculate statistics.

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)].

Returns:
amp_mediansdict

Median values for each amp. Keys are ramp numbers as strings with even/odd designation (e.g. '1_even').

identify_tables()[source]

Determine which database tables to use for a run of the bias monitor.

image_to_png(image, outname)[source]

Ouputs an image array into a png file.

Parameters:
imagenumpy.ndarray

2D image array.

outnamestr

The name given to the output png file.

Returns:
output_filenamestr

The full path to the output png file.

make_histogram(data)[source]

Creates a histogram of the input data and returns the bin centers and the counts in each bin.

Parameters:
datanumpy.ndarray

The input data.

Returns:
countsnumpy.ndarray

The counts in each histogram bin.

bin_centersnumpy.ndarray

The histogram bin centers.

Query the query history database and return the information on the most recent query for the given aperture_name where the bias monitor was executed.

Returns:
query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the bias monitor was run.

process(file_list)[source]

The main method for processing darks. See module docstrings for further details.

Parameters:
file_listlist

List of filenames (including full paths) to the dark current files.

run()[source]

The main method. See module docstrings for further details.

dark_monitor.py

This module contains code for the dark current monitor, which performs some basic analysis to check whether the dark current behavior for the most recent set of input files is consistent with that from past files.

If enough new files for a given instrument/aperture combination (currently the files must be identified as dark current files in the exp_type header keyword) are present in the filesystem at the time the dark_monitor is called, the files are first run through the the appropriate pipeline steps to produce slope images.

A mean slope image as well as a standard deviation slope image is created by sigma-clipping on a pixel by pixel basis. The mean and standard deviation images are saved to a fits file, the name of which is entered into the <Instrument>DarkCurrent database table.

The mean slope image is then normalized by an existing baseline slope image. New hot pixels are identified as those with normalized signal rates above a hot_threshold value. Similarly, pixels with normalized signal rates below a dead_threshold are flagged as new dead pixels.

The standard deviation slope image is normalized by a baseline (historical) standard deviation image. Pixels with normalized values above a noise threshold are flagged as newly noisy pixels.

New hot, dead, and noisy pixels are saved to the DarkPixelStats database table.

Next, the dark current in the mean slope image is examined. A histogram of the slope values is created for the pixels in each amplifier, as well as for all pixels on the detector. In all cases, a Gaussian is fit to the histogram. Currently for NIRCam and NIRISS, a double Gaussian is also fit to the histogram from the entire detector.

The histogram itself as well as the best-fit Gaussian and double Gaussian parameters are saved to the DarkDarkCurrent database table.

Author

  • Bryan Hilbert

Use

This module can be used from the command line as such:

python dark_monitor.py
class jwql.instrument_monitors.common_monitors.dark_monitor.Dark[source]

Class for executing the dark current monitor.

This class will search for new (since the previous instance of the class) dark current files in the file system. It will loop over instrument/aperture combinations and find the number of new dark current files available. If there are enough, it will copy the files over to a working directory and run the monitor. This will create a mean dark current rate image, create a histogram of the dark current values, and fit several functions to the histogram. It will also compare the dark current image to a historical image in order to search for new hot or dead pixels. Results are all saved to database tables.

Parameters:
testingbool

For pytest. If True, an instance of Dark is created, but no other code is executed.

Raises:
ValueError

If encountering an unrecognized bad pixel type

ValueError

If the most recent query search returns more than one entry

Attributes:
output_dirstr

Path into which outputs will be placed

data_dirstr

Path into which new dark files will be copied to be worked on

query_startfloat

MJD start date to use for querying MAST

query_endfloat

MJD end date to use for querying MAST

instrumentstr

Name of instrument used to collect the dark current data

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL)

query_tablesqlalchemy table

Table containing the history of dark current queries to MAST for each instrument/aperture combination

pixel_tablesqlalchemy table

Table containing lists of hot/dead/noisy pixels found for each instrument/detector

stats_tablesqlalchemy table

Table containing dark current analysis results. Mean/stdev values, histogram information, Gaussian fitting results, etc.

add_bad_pix(coordinates, pixel_type, files, mean_filename, baseline_filename, observation_start_time, observation_mid_time, observation_end_time)[source]

Add a set of bad pixels to the bad pixel database table

Parameters:
coordinatestuple

Tuple of two lists, containing x,y coordinates of bad pixels (Output of np.where call)

pixel_typestr

Type of bad pixel. Options are dead, hot, and noisy

fileslist

List of fits files which were used to identify the bad pixels

mean_filenamestr

Name of fits file containing the mean dark rate image used to find these bad pixels

baseline_filenamestr

Name of fits file containing the baseline dark rate image used to find these bad pixels

observation_start_timedatetime.datetime

Observation time of the earliest file in files

observation_mid_timedatetime.datetime

Average of the observation times in files

observation_end_timedatetime.datetime

Observation time of the latest file in files

exclude_existing_badpix(badpix, pixel_type)[source]

Given a set of coordinates of bad pixels, determine which of these pixels have been previously identified and remove them from the list

Parameters:
badpixtuple

Tuple of lists containing x and y pixel coordinates. (Output of numpy.where call)

pixel_typestr

Type of bad pixel being examined. Options are hot, dead, and noisy

Returns:
new_pixels_xlist

List of x coordinates of new bad pixels

new_pixels_ylist

List of y coordinates of new bad pixels

find_hot_dead_pixels(mean_image, comparison_image, hot_threshold=2.0, dead_threshold=0.1)[source]

Create the ratio of the slope image to a baseline slope image. Pixels in the ratio image with values above hot_threshold will be marked as hot, and those with ratio values less than dead_threshold will be marked as dead.

Parameters:
mean_imagenumpy.ndarray

2D array containing the slope image from the new data

comparison_imagenumpy.ndarray

2D array containing the baseline slope image to compare against the new slope image.

hot_thresholdfloat

(mean_image / comparison_image) ratio value above which a pixel is considered hot.

dead_thresholdfloat

(mean_image / comparison_image) ratio value below which a pixel is considered dead.

Returns:
hotpixtuple

Tuple (of lists) containing x,y coordinates of newly hot pixels

deadpixtuple

Tuple (of lists) containing x,y coordinates of newly dead pixels

get_baseline_filename()[source]

Query the database and return the filename of the baseline (comparison) mean dark slope image to use when searching for new hot/dead/noisy pixels. For this we assume that the most recent baseline file for the given detector is the one to use.

Returns:
filenamestr

Name of fits file containing the baseline image

get_metadata(filename)[source]

Collect basic metadata from a fits file

Parameters:
filenamestr

Name of fits file to examine

identify_tables()[source]

Determine which database tables to use for a run of the dark monitor

Query the query history database and return the information on the most recent query for the given aperture_name where the dark monitor was executed.

Returns:
query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the dark monitor was run.

noise_check(new_noise_image, baseline_noise_image, threshold=1.5)[source]

Create the ratio of the stdev (noise) image to a baseline noise image. Pixels in the ratio image with values above threshold will be marked as newly noisy.

Parameters:
new_noise_imagenumpy.ndarray

2D array containing the noise image from the new data

baseline_noise_imagenumpy.ndarray

2D array containing the baseline noise image to compare against the new noise image.

thresholdfloat

(new_noise_image / baseline_noise_image) ratio value above which a pixel is considered newly noisey.

Returns:
noisytuple

Tuple (of lists) of x,y coordinates of newly noisy pixels

process(file_list)[source]

The main method for processing darks. See module docstrings for further details.

Parameters:
file_listlist

List of filenames (including full paths) to the dark current files

read_baseline_slope_image(filename)[source]

Read in a baseline mean slope image and associated standard deviation image from the given fits file

Parameters:
filenamestr

Name of fits file to be read in

Returns:
mean_imagenumpy.ndarray

2D mean slope image

stdev_imagenumpy.ndarray

2D stdev image

run()[source]

The main method. See module docstrings for further details.

save_mean_slope_image(slope_img, stdev_img, files)[source]

Save the mean slope image and associated stdev image to a file

Parameters:
slope_imgnumpy.ndarray

2D array containing the mean slope image

stdev_imgnumpy.ndarray

2D array containing the stdev image associated with the mean slope image.

fileslist

List of input files used to construct the mean slope image

Returns:
output_filenamestr

Name of fits file to save mean and stdev images within

shift_to_full_frame(coords)[source]

Shift the input list of pixels from the subarray coordinate system to the full frame coordinate system

Parameters:
coordstup

(x, y) pixel coordinates in subarray coordinate system

Returns:
coordstup

(x, y) pixel coordinates in full frame coordinate system

stats_by_amp(image, amps)[source]

Calculate statistics in the input image for each amplifier as well as the full image

Parameters:
imagenumpy.ndarray

2D array on which to calculate statistics

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)]

Returns:
amp_meansdict

Sigma-clipped mean value for each amp. Keys are amp numbers as strings (e.g. '1')

amp_stdevsdict

Sigma-clipped standard deviation for each amp. Keys are amp numbers as strings (e.g. '1')

gaussian_paramsdict

Best-fit Gaussian parameters to the dark current histogram. Keys are amp numbers as strings. Values are three-element lists [amplitude, peak, width]. Each element in the list is a tuple of the best-fit value and the associated uncertainty.

gaussian_chi_squareddict

Reduced chi-squared for the best-fit parameters. Keys are amp numbers as strings

double_gaussian_paramsdict

Best-fit double Gaussian parameters to the dark current histogram. Keys are amp numbers as strings. Values are six- element lists. (3-elements * 2 Gaussians). [amplitude1, peak1, stdev1, amplitude2, peak2, stdev2] Each element of the list is a tuple containing the best-fit value and associated uncertainty.

double_gaussian_chi_squareddict

Reduced chi-squared for the best-fit parameters. Keys are amp numbers as strings

histnumpy.ndarray

1D array of histogram values

bin_centersnumpy.ndarray

1D array of bin centers that match the hist values.

jwql.instrument_monitors.common_monitors.dark_monitor.mast_query_darks(instrument, aperture, start_date, end_date, readpatt=None)[source]

Use astroquery to search MAST for dark current data

Parameters:
instrumentstr

Instrument name (e.g. nircam)

aperturestr

Detector aperture to search for (e.g. NRCA1_FULL)

start_datefloat

Starting date for the search in MJD

end_datefloat

Ending date for the search in MJD

readpattstr

Readout pattern to search for (e.g. RAPID). If None, readout pattern will not be added to the query parameters.

Returns:
query_resultslist

List of dictionaries containing the query results

readnoise_monitor.py

This module contains code for the readnoise monitor, which monitors the readnoise levels in dark exposures as well as the accuracy of the pipeline readnoise reference files over time.

For each instrument, the readnoise, technically the correlated double sampling (CDS) noise, is found by calculating the standard deviation through a stack of consecutive frame differences in each dark exposure. The sigma-clipped mean and standard deviation in each of these readnoise images, as well as histogram distributions, are recorded in the <Instrument>ReadnoiseStats database table.

Next, each of these readnoise images are differenced with the current pipeline readnoise reference file to identify the need for new reference files. A histogram distribution of these difference images, as well as the sigma-clipped mean and standard deviation, are recorded in the <Instrument>ReadnoiseStats database table. A png version of these difference images is also saved for visual inspection.

Author

  • Ben Sunnquist

Use

This module can be used from the command line as such:

python readnoise_monitor.py
class jwql.instrument_monitors.common_monitors.readnoise_monitor.Readnoise[source]

Class for executing the readnoise monitor.

This class will search for new dark current files in the file system for each instrument and will run the monitor on these files. The monitor will create a readnoise image for each of the new dark files. It will then perform statistical measurements on these readnoise images, as well as their differences with the current pipeline readnoise reference file, in order to monitor the readnoise levels over time as well as ensure the pipeline readnoise reference file is sufficiently capturing the current readnoise behavior. Results are all saved to database tables.

Attributes:
output_dirstr

Path into which outputs will be placed.

data_dirstr

Path into which new dark files will be copied to be worked on.

query_startfloat

MJD start date to use for querying MAST.

query_endfloat

MJD end date to use for querying MAST.

instrumentstr

Name of instrument used to collect the dark current data.

aperturestr

Name of the aperture used for the dark current (e.g. NRCA1_FULL).

determine_pipeline_steps()[source]

Determines the necessary JWST pipelines steps to run on a given dark file.

Returns:
pipeline_stepscollections.OrderedDict

The pipeline steps to run.

file_exists_in_database(filename)[source]

Checks if an entry for filename exists in the readnoise stats database.

Parameters:
filenamestr

The full path to the uncal filename.

Returns:
file_existsbool

True if filename exists in the readnoise stats database.

get_amp_stats(image, amps)[source]

Calculates the sigma-clipped mean and stddev, as well as the histogram stats in the input image for each amplifier.

Parameters:
imagenumpy.ndarray

2D array on which to calculate statistics.

ampsdict

Dictionary containing amp boundary coordinates (output from amplifier_info function) amps[key] = [(xmin, xmax, xstep), (ymin, ymax, ystep)]

Returns:
amp_statsdict

Contains the image statistics for each amp.

get_metadata(filename)[source]

Collect basic metadata from a fits file.

Parameters:
filenamestr

Name of fits file to examine.

identify_tables()[source]

Determine which database tables to use for a run of the readnoise monitor.

image_to_png(image, outname)[source]

Outputs an image array into a png file.

Parameters:
imagenumpy.ndarray

2D image array.

outnamestr

The name given to the output png file.

Returns:
output_filenamestr

The full path to the output png file.

make_crds_parameter_dict()[source]

Construct a paramter dictionary to be used for querying CRDS for the current reffiles in use by the JWST pipeline.

Returns:
parametersdict

Dictionary of parameters, in the format expected by CRDS.

make_histogram(data)[source]

Creates a histogram of the input data and returns the bin centers and the counts in each bin.

Parameters:
datanumpy.ndarray

The input data.

Returns:
countsnumpy.ndarray

The counts in each histogram bin.

bin_centersnumpy.ndarray

The histogram bin centers.

make_readnoise_image(data)[source]

Calculates the readnoise for the given input dark current ramp.

Parameters:
datanumpy.ndarray

The input ramp data. The data shape is assumed to be a 4D array in DMS format (integration, group, y, x).

Returns:
readnoisenumpy.ndarray

The 2D readnoise image.

Query the query history database and return the information on the most recent query for the given aperture_name where the readnoise monitor was executed.

Returns:
query_resultfloat

Date (in MJD) of the ending range of the previous MAST query where the readnoise monitor was run.

process(file_list)[source]

The main method for processing darks. See module docstrings for further details.

Parameters:
file_listlist

List of filenames (including full paths) to the dark current files.

run()[source]

The main method. See module docstrings for further details.