TolTECA Documentation

tolteca is a Python package developed to handle data taken by the TolTEC Camera on the Large Millimeter Telescope.

While TolTEC is its main target instrument, the infrastructure and generic workflow are implemented in a instrument agnostic fashion, which allows the functionalities to be extendable to other instruments.

Getting Started

Todo

Add details.

Using tolteca

Overview of Workflow

• RuntimeContext and RuntimeBase

• Commandline Interface

• Programming Interface

Data File and Data Product Management

Data Reduction Pipeline

Observation Simulator

Web-based Tools

• Web-based Tools (tolteca.web)
  • Introduction
  • Getting Started

Recipes

• TolTECA Recipes
  • tolteca.recipes Package
  • tolteca.recipes.autodrive Module
  • tolteca.recipes.dataset_rsync Module

Scripts

Reference/API

tolteca Package

tolteca.cli Package

Functions

main([args])

The CLI entry point.

tolteca.common.toltec Package

tolteca.common.lmt Package

tolteca.datamodels.toltec Package

This module implements the TolTEC data model.

The TolTEC data model covers two topics:

1. Basic Obs Data (BOD)

   Basic Obs Data are the data from the telescope/instrument control system (TCS/ICS) and the timely analysis service (TAS).

   These data are typically stored on disk on a per-interface basis, and carry full runtime information of the TolTEC instrument.

   The name “basic obs” (BO) refers to a continuous file-writing transaction executed by the TCS/ICS. This is different from an “obs request” (OR) which may consist of multiple BOs.

   Each basic obs is assigned a unique transaction identifier, consists of a 4-tuple of “master”, “obsnum”, “subobsnum”, and “scannum”. In addition, the basic obs data stream may be relayed (duplicated) to remote destinations and files will be created with the destination local repeat writers. The (repeated) copy of the data files are distinguished by an integer field “repeat”. The original data has “repeat” value of 0.

   The TAS monitors newly generated raw obs data and runs various data reduction recipes to create timely view data products to guide the observation. One of the most important output of this is noted as the basic reduced data files for each input raw obs data files.

   Both raw obs data and basic reduced data can be handled by the wrapper class BasicObsData. By providing a unified interface for access the meta data and data in the data files, this wrapper class hides away the details of handling of interacting with each data storage format for different data kinds, which is implemented in the various subclasses of DataFileIO defined in toltec.

   More over, the helper class BasicObsDataset provides a table-like way to work with a collection of BOD files for carrying out instrument level diagnostics and data analysis.

2. Data Products and the Associations

   Data Products (DPs) and Data Product Associations (DPAs) are the core concepts of the TolTEC data management system, which involves both the raw data and the reduced data.

   The DP and DPA information are managed using the data product database (DPDB). The modules db, dp, and toltecdp implement the database structure, the data product specification/schema, as well as the interface to manage the DP and DPAs.

   Data analysis tasks run via TolTECA read and write to DPDB to record the inputs, outputs, the associations, as well the runtime context info.

   The key element of a data product is an index object (DPIndex), typically stored along with the actual (textual or binary) data files as an standalone YAML (or JSON) file.

   One of the key property of a data product is the data product type. The contents of the DPIndex for different data types are constrained by the corresponding schemas.

   Some data product types are:

   • dp_raw_obs

     Simple collection of raw basic obs data files (BODs).

   • dp_basic_reduced_obs

     Simple collection of basic reduced obs data generated by the TAS, one-to-one mapping to the dp_raw_obs.

   • dp_science

     Reduced, science-ready maps/catalogs.

   The DPs are related to each other through association info noted as DPAs. Each DPA typically has contextual data (meta data) to describe the relations of the DPs.

Classes

BasicObsData(source[, open_])	A class that provides unified IO interface to TolTEC basic obs data.
BasicObsDataset([index_table, bod_list, ...])	A helper class to access a set of TolTEC basic obs data items.
ToltecDataProd()	This class provides access to TolTEC data product info.

Class Inheritance Diagram

tolteca.datamodels.fs Package

tolteca.datamodels.db Package

tolteca.datamodels.io Package

Functions

open_file(source[, format])

A alias of open

Classes

KidsModelParamsIO([source, open_])	"A class to read KIDs model parameters.
NcFileIO([source, open_, load_meta_on_open, ...])	A class to read data from TolTEC netCDF files.

Class Inheritance Diagram

tolteca.reduce Package

Classes

ReduConfig(jobkey, inputs, steps)	The config for tolteca.reduce.
PipelineRuntime(config)	A class to run the data reduction pipeline.
PipelineRuntimeError	Raise when errors occur in PipelineRuntime.

Class Inheritance Diagram

tolteca.reduce.engines Package

Classes

BasicObsDataset([index_table, bod_list, ...])	A helper class to access a set of TolTEC basic obs data items.
Citlali([path, version, use_env_path])	A wrapper class of Citlali, the TolTEC data reduction pipeline engine.
CitlaliConfig(low_level, pathlib.Path, ...)	The high-level config for Citlali.
CitlaliStepConfig(**kwargs)	The config class for reduction with Citlali.
DataclassNamespace	A subclass of tollan.utils.namespace.Namespace that support dataclass type members.
Optional(*args, **kwargs)	Marker for an optional part of the validation Schema.
Or(*args, **kwargs)	Utility function to combine validation directives in a OR Boolean fashion.
RelPathSchema([check_exists])
Schema(schema[, error, ignore_extra_keys, ...])	Entry point of the library, use this class to instantiate validation schema for the data that will be validated.
log_to_file(filepath[, logger, level, ...])	A context manager to log to file.

Class Inheritance Diagram

tolteca.reduce.toltec Package

Classes

BasicObsDataset([index_table, bod_list, ...])	A helper class to access a set of TolTEC basic obs data items.
BasicObsDatasetLoader()	A class to collect TolTEC basic obs dataset from local files.
LocalFileDataStore	A class describing data managed in local file system.
NamespaceMixin()	A mixin class that allows converting object to and from dict.

Class Inheritance Diagram

tolteca.simu Package

Classes

SimuConfig(jobkey, instrument, mapping, ...)	The config for tolteca.simu.
SimulatorRuntime(config)	A class to run the simulator.
SimulatorRuntimeError	Raise when errors occur in SimulatorRuntime.

Class Inheritance Diagram

tolteca.simu.base Module

Classes

ProjModel(*args, **kwargs)	Base class for models that transform properties from one frame to another.
LabelFrame(axes_names[, axes_order, name])	An coordinate frame for describing discrete detector attributes.

Class Inheritance Diagram

tolteca.simu.mapping Package

Classes

PatternKind(value)	The available pattern kinds.
LmtTcsTrajMappingModel(filepath)	The class for to describe trajectories loaded from LMT TCS tel.nc file.

Class Inheritance Diagram

tolteca.simu.sources Package

Classes

CatalogSourceModel(catalog[, pos_cols, ...])	The class for simulator source specified by catalog.
DataclassNamespace	A subclass of tollan.utils.namespace.Namespace that support dataclass type members.
DictTemplateListSchema(template_key[, ...])	A schema that can resolve list of dict specified with templates.
ImageSourceConfig	The config class for image source model created from FITS image file.
ImageSourceModel(hdulist[, data_exts])	The class for simulator source from FITS image.
Literal(value[, description])
Optional(*args, **kwargs)	Marker for an optional part of the validation Schema.
Or(*args, **kwargs)	Utility function to combine validation directives in a OR Boolean fashion.
PointSourceCatalogSourceConfig	The config class for point source catalog source model created from catalog file.
RelPathSchema([check_exists])
Schema(schema[, error, ignore_extra_keys, ...])	Entry point of the library, use this class to instantiate validation schema for the data that will be validated.

Class Inheritance Diagram

tolteca.simu.exports Package

Classes

DataclassNamespace	A subclass of tollan.utils.namespace.Namespace that support dataclass type members.
LmtOTComposer()	A class to create LMT OT scripts.
LmtOTExporterConfig(*args, **kwargs)	The config class for LMT OT script exporter.
Optional(*args, **kwargs)	Marker for an optional part of the validation Schema.
Schema(schema[, error, ignore_extra_keys, ...])	Entry point of the library, use this class to instantiate validation schema for the data that will be validated.
datetime(year, month, day[, hour[, minute[, ...)	The year, month and day arguments are required.
timezone	Fixed offset from UTC implementation of tzinfo.

Class Inheritance Diagram

tolteca.simu.plots Package

Functions

plot_mapping(simulator, mapping[, ...])
plot_visibility(t0, targets[, target_names, ...])

Classes

DataclassNamespace	A subclass of tollan.utils.namespace.Namespace that support dataclass type members.
MappingPlotConfig	The config class for mapping trajectory plot.
Optional(*args, **kwargs)	Marker for an optional part of the validation Schema.
Schema(schema[, error, ignore_extra_keys, ...])	Entry point of the library, use this class to instantiate validation schema for the data that will be validated.
VisibilityPlotConfig	The config class for visibility plot.

Class Inheritance Diagram

tolteca.simu.toltec Package

Classes

ToltecObsSimulatorConfig(**kwargs)	The config class for TolTEC observation simulator.
ToltecHwpConfig(f_rot, f_smp, rotator_enabled)	The config class for TolTEC half-wave plate and the rotator.
ToltecPowerLoadingModelConfig(*args, **kwargs)	The config class to compute the TolTEC detector power loading.

Class Inheritance Diagram

tolteca.utils Package

Functions

get_user_data_dir()
get_pkg_data_path()	Return the package data path.
yaml_load(stream)
yaml_dump(config[, output])	Dump config as YAML to output
dict_from_cli_args(args)	Return a nested dict composed from CLI arguments.

Classes

RuntimeContext(config)	A class to manage configurations.
RuntimeContextError
RuntimeBase(config)	A base class that consumes RuntimeContext.
RuntimeBaseError
ConfigLoaderError	Raise when error in ConfigLoader.
ConfigLoader([files, load_sys_config, ...])	A helper class to load config files for tolteca.

Class Inheritance Diagram

tolteca.web Package

Classes

WebConfig(apps)	The config for tolteca.web.
WebRuntime(config)	A class that manages the runtime context of web apps.
WebRuntimeError

Class Inheritance Diagram

tolteca.web.apps Package