database

database_interface.py

A module to interact with the JWQL postgresql database jwqldb

The load_connection() function within this module allows the user to connect to the jwqldb database via the session, base, and engine objects (described below). The classes within serve as ORMs (Object-relational mappings) that define the individual tables of the relational database.

The engine object serves as the low-level database API and perhaps most importantly contains dialects which allows the sqlalchemy module to communicate with the database.

The base object serves as a base class for class definitions. It produces Table objects and constructs ORMs.

The session object manages operations on ORM-mapped objects, as construced by the base. These operations include querying, for example.

Authors

  • Joe Filippazzo

  • Johannes Sahlmann

  • Matthew Bourque

  • Lauren Chambers

  • Bryan Hilbert

  • Misty Cracraft

  • Sara Ogaz

  • Maria Pena-Guerrero

Use

Executing the module on the command line will build the database tables defined within:

python database_interface.py

Users wishing to interact with the existing database may do so by importing various connection objects and database tables, for example:

from jwql.database.database_interface import Anomaly
from jwql.database.database_interface import session

results = session.query(Anomaly).all()

Dependencies

The user must have a configuration file named config.json placed in the jwql directory.

class jwql.database.database_interface.CentralStore(**kwargs)[source]

ORM for the central storage area filesystem monitor table

area
available
date
id
size
used
jwql.database.database_interface.FGSAnomaly

alias of fgs_anomaly

jwql.database.database_interface.FGSBadPixelQueryHistory

alias of fgs_bad_pixel_query_history

jwql.database.database_interface.FGSBadPixelStats

alias of fgs_bad_pixel_stats

jwql.database.database_interface.FGSCosmicRayQueryHistory

alias of fgs_cosmic_ray_query_history

jwql.database.database_interface.FGSCosmicRayStats

alias of fgs_cosmic_ray_stats

jwql.database.database_interface.FGSDarkDarkCurrent

alias of fgs_dark_dark_current

jwql.database.database_interface.FGSDarkPixelStats

alias of fgs_dark_pixel_stats

jwql.database.database_interface.FGSDarkQueryHistory

alias of fgs_dark_query_history

jwql.database.database_interface.FGSEDBBlockStats

alias of fgs_edb_blocks_stats

jwql.database.database_interface.FGSEDBDailyStats

alias of fgs_edb_daily_stats

jwql.database.database_interface.FGSEDBEveryChangeStats

alias of fgs_edb_every_change_stats

jwql.database.database_interface.FGSEDBTimeIntervalStats

alias of fgs_edb_time_interval_stats

jwql.database.database_interface.FGSReadnoiseQueryHistory

alias of fgs_readnoise_query_history

jwql.database.database_interface.FGSReadnoiseStats

alias of fgs_readnoise_stats

class jwql.database.database_interface.FilesystemCharacteristics(**kwargs)[source]

ORM for table containing instrument-specific lists of the number of obervations corresponding to various instrument characteristics (e.g. filters)

date
filter_pupil
id
instrument
obs_per_filter_pupil
class jwql.database.database_interface.FilesystemGeneral(**kwargs)[source]

ORM for the general (non instrument specific) filesystem monitor table

available
date
fits_file_count
fits_file_size
id
total_file_count
total_file_size
used
class jwql.database.database_interface.FilesystemInstrument(**kwargs)[source]

ORM for the instrument specific filesystem monitor table

property colnames

A list of all column names in this table EXCEPT the date column

count
date
filetype
id
instrument
size
jwql.database.database_interface.MIRIAnomaly

alias of miri_anomaly

jwql.database.database_interface.MIRIBadPixelQueryHistory

alias of miri_bad_pixel_query_history

jwql.database.database_interface.MIRIBadPixelStats

alias of miri_bad_pixel_stats

jwql.database.database_interface.MIRICosmicRayQueryHistory

alias of miri_cosmic_ray_query_history

jwql.database.database_interface.MIRICosmicRayStats

alias of miri_cosmic_ray_stats

jwql.database.database_interface.MIRIDarkDarkCurrent

alias of miri_dark_dark_current

jwql.database.database_interface.MIRIDarkPixelStats

alias of miri_dark_pixel_stats

jwql.database.database_interface.MIRIDarkQueryHistory

alias of miri_dark_query_history

jwql.database.database_interface.MIRIEDBBlockStats

alias of miri_edb_blocks_stats

jwql.database.database_interface.MIRIEDBDailyStats

alias of miri_edb_daily_stats

jwql.database.database_interface.MIRIEDBEveryChangeStats

alias of miri_edb_every_change_stats

jwql.database.database_interface.MIRIEDBTimeIntervalStats

alias of miri_edb_time_interval_stats

jwql.database.database_interface.MIRIReadnoiseQueryHistory

alias of miri_readnoise_query_history

jwql.database.database_interface.MIRIReadnoiseStats

alias of miri_readnoise_stats

class jwql.database.database_interface.Monitor(**kwargs)[source]

ORM for the monitor table

end_time
id
log_file
monitor_name
start_time
status
jwql.database.database_interface.NIRCamAnomaly

alias of nircam_anomaly

jwql.database.database_interface.NIRCamBadPixelQueryHistory

alias of nircam_bad_pixel_query_history

jwql.database.database_interface.NIRCamBadPixelStats

alias of nircam_bad_pixel_stats

jwql.database.database_interface.NIRCamBiasQueryHistory

alias of nircam_bias_query_history

jwql.database.database_interface.NIRCamBiasStats

alias of nircam_bias_stats

jwql.database.database_interface.NIRCamClawQueryHistory

alias of nircam_claw_query_history

jwql.database.database_interface.NIRCamClawStats

alias of nircam_claw_stats

jwql.database.database_interface.NIRCamCosmicRayQueryHistory

alias of nircam_cosmic_ray_query_history

jwql.database.database_interface.NIRCamCosmicRayStats

alias of nircam_cosmic_ray_stats

jwql.database.database_interface.NIRCamDarkDarkCurrent

alias of nircam_dark_dark_current

jwql.database.database_interface.NIRCamDarkPixelStats

alias of nircam_dark_pixel_stats

jwql.database.database_interface.NIRCamDarkQueryHistory

alias of nircam_dark_query_history

jwql.database.database_interface.NIRCamEDBBlockStats

alias of nircam_edb_blocks_stats

jwql.database.database_interface.NIRCamEDBDailyStats

alias of nircam_edb_daily_stats

jwql.database.database_interface.NIRCamEDBEveryChangeStats

alias of nircam_edb_every_change_stats

jwql.database.database_interface.NIRCamEDBTimeIntervalStats

alias of nircam_edb_time_interval_stats

jwql.database.database_interface.NIRCamReadnoiseQueryHistory

alias of nircam_readnoise_query_history

jwql.database.database_interface.NIRCamReadnoiseStats

alias of nircam_readnoise_stats

jwql.database.database_interface.NIRISSAnomaly

alias of niriss_anomaly

jwql.database.database_interface.NIRISSBadPixelQueryHistory

alias of niriss_bad_pixel_query_history

jwql.database.database_interface.NIRISSBadPixelStats

alias of niriss_bad_pixel_stats

jwql.database.database_interface.NIRISSBiasQueryHistory

alias of niriss_bias_query_history

jwql.database.database_interface.NIRISSBiasStats

alias of niriss_bias_stats

jwql.database.database_interface.NIRISSCosmicRayQueryHistory

alias of niriss_cosmic_ray_query_history

jwql.database.database_interface.NIRISSCosmicRayStats

alias of niriss_cosmic_ray_stats

jwql.database.database_interface.NIRISSDarkDarkCurrent

alias of niriss_dark_dark_current

jwql.database.database_interface.NIRISSDarkPixelStats

alias of niriss_dark_pixel_stats

jwql.database.database_interface.NIRISSDarkQueryHistory

alias of niriss_dark_query_history

jwql.database.database_interface.NIRISSEDBBlockStats

alias of niriss_edb_blocks_stats

jwql.database.database_interface.NIRISSEDBDailyStats

alias of niriss_edb_daily_stats

jwql.database.database_interface.NIRISSEDBEveryChangeStats

alias of niriss_edb_every_change_stats

jwql.database.database_interface.NIRISSEDBTimeIntervalStats

alias of niriss_edb_time_interval_stats

jwql.database.database_interface.NIRISSReadnoiseQueryHistory

alias of niriss_readnoise_query_history

jwql.database.database_interface.NIRISSReadnoiseStats

alias of niriss_readnoise_stats

jwql.database.database_interface.NIRSpecAnomaly

alias of nirspec_anomaly

jwql.database.database_interface.NIRSpecBadPixelQueryHistory

alias of nirspec_bad_pixel_query_history

jwql.database.database_interface.NIRSpecBadPixelStats

alias of nirspec_bad_pixel_stats

jwql.database.database_interface.NIRSpecBiasQueryHistory

alias of nirspec_bias_query_history

jwql.database.database_interface.NIRSpecBiasStats

alias of nirspec_bias_stats

jwql.database.database_interface.NIRSpecCosmicRayQueryHistory

alias of nirspec_cosmic_ray_query_history

jwql.database.database_interface.NIRSpecCosmicRayStats

alias of nirspec_cosmic_ray_stats

jwql.database.database_interface.NIRSpecDarkDarkCurrent

alias of nirspec_dark_dark_current

jwql.database.database_interface.NIRSpecDarkPixelStats

alias of nirspec_dark_pixel_stats

jwql.database.database_interface.NIRSpecDarkQueryHistory

alias of nirspec_dark_query_history

jwql.database.database_interface.NIRSpecEDBBlockStats

alias of nirspec_edb_blocks_stats

jwql.database.database_interface.NIRSpecEDBDailyStats

alias of nirspec_edb_daily_stats

jwql.database.database_interface.NIRSpecEDBEveryChangeStats

alias of nirspec_edb_every_change_stats

jwql.database.database_interface.NIRSpecEDBTimeIntervalStats

alias of nirspec_edb_time_interval_stats

jwql.database.database_interface.NIRSpecReadnoiseQueryHistory

alias of nirspec_readnoise_query_history

jwql.database.database_interface.NIRSpecReadnoiseStats

alias of nirspec_readnoise_stats

jwql.database.database_interface.NIRSpecTAQueryHistory

alias of nirspec_ta_query_history

jwql.database.database_interface.NIRSpecTAStats

alias of nirspec_ta_stats

jwql.database.database_interface.anomaly_orm_factory(class_name)[source]

Create a SQLAlchemy ORM Class for an anomaly table.

Parameters:
class_namestr

The name of the class to be created

Returns:
classobj

The SQLAlchemy ORM

jwql.database.database_interface.get_monitor_columns(data_dict, table_name)[source]

Read in the corresponding table definition text file to generate SQLAlchemy columns for the table.

Parameters:
data_dictdict

A dictionary whose keys are column names and whose values are column definitions.

table_namestr

The name of the database table

Returns:
data_dictdict

An updated data_dict with the approriate columns for the monitor added.

jwql.database.database_interface.get_monitor_table_constraints(data_dict, table_name)[source]

Add any necessary table constrains to the given table via the data_dict.

Parameters:
data_dictdict

A dictionary whose keys are column names and whose values are column definitions.

table_namestr

The name of the database table

Returns:
data_dictdict

An updated data_dict with the approriate table constraints for the monitor added.

jwql.database.database_interface.get_unique_values_per_column(table, column_name)[source]

Return a list of the unique values from a particular column in the given table.

Parameters:
tablesqlalchemy.orm.decl_api.DeclarativeMeta

SQL table to be searched. (e.g. table = eval(‘NIRCamDarkPixelStats’))

column_namestr

Column name within the table to query

Returns:
distinct_colvalslist

List of unique values in the given column

jwql.database.database_interface.load_connection(connection_string)[source]

Return session, base, engine, and metadata objects for connecting to the jwqldb database.

Create an engine using an given connection_string. Create a base class and session class from the engine. Create an instance of the session class. Return the session, base, and engine instances. This was stolen from the ascql repository.

Parameters:
connection_stringstr

A postgresql database connection string. The connection string should take the form: dialect+driver://username:password@host:port/database

Returns:
sessionsesson object

Provides a holding zone for all objects loaded or associated with the database.

basebase object

Provides a base class for declarative class definitions.

engineengine object

Provides a source of database connectivity and behavior.

meta: metadata object

The connection metadata

References

ascql:

https://github.com/spacetelescope/acsql/blob/master/acsql/database/database_interface.py

jwql.database.database_interface.monitor_orm_factory(class_name)[source]

Create a SQLAlchemy ORM Class for a jwql instrument monitor.

Parameters:
class_namestr

The name of the class to be created

Returns:
classobj

The SQLAlchemy ORM

reset_database.py

Reset all tables in the jwqldb database.

Authors

  • Matthew Bourque

Use

This script is intended to be used in the command line:

python reset_database.py

Dependencies

Users must have a config.json configuration file with a proper connection_string key that points to the jwqldb database. The connection_string format is postgresql+psycopg2://user:password@host:port/database.