giuseppe@125: SCC products giuseppe@125: ============ giuseppe@125: giuseppe@125: giuseppe@125: Introduction giuseppe@125: ------------ giuseppe@125: giuseppe@125: The Single Calculus Chain (SCC) is the standard EARLINET tool to perform giuseppe@125: automatic and quality checked analysis of raw lidar data. It is giuseppe@125: composed by the following modules: giuseppe@125: giuseppe@125: - HiRELPP (High Resolution ELPP) giuseppe@126: - CloudScreen (SCC cloud screen module) giuseppe@125: - ELPP (EARLINET Lidar Pre-Processor) giuseppe@125: - ELDA (EARLINET Lidar Data Analizer) giuseppe@125: - ELDEC (EARLINET Lidar DEpolarization Calibrator) giuseppe@125: - ELIC (EARLINET LIdar Calibrator) giuseppe@125: - ELQUICK (EARLINET Lidar QUICJKlook) giuseppe@125: giuseppe@125: giuseppe@125: HiRELPP giuseppe@125: -------- giuseppe@125: giuseppe@125: The HiRELPP module implements the corrections to be applied to the raw lidar signals giuseppe@125: before they can be used to derive higher level products. giuseppe@125: All the operations implemented in HiRELPP are designed to preserve both the vertical and time resolution as high as possible. giuseppe@125: Some instrumental effects (like for example, dead-time correction, trigger-delay correction, overlap correction, giuseppe@125: atmospheric and electronic background subtraction, low- and high-range automatic signal glueing) giuseppe@125: are corrected following the recommendations provided by the EARLINET quality assurance program. giuseppe@125: giuseppe@125: Dead-time correction giuseppe@125: #################### giuseppe@125: giuseppe@125: The dead-time corresponds to a maximum count rate. The dead- time causes a non-linearity giuseppe@125: between the actual intensity at the photo-multiplier photocathode and the counted events, giuseppe@125: which can be described theoretically by means of photon statistics. Actual detector can be giuseppe@125: modelled as the paralyzable and the non-paralyzable model. Once information about the model giuseppe@125: to use for describing the counting system and the dead-time value is determined, based on giuseppe@125: standard operating procedures defined by the ACTRIS Center for Aerosol remote Sensing), these giuseppe@125: are provided to HiRELPP and the acquired counts are corrected for the dead-time effect. giuseppe@125: giuseppe@125: giuseppe@125: Trigger-delay correction giuseppe@125: ######################## giuseppe@125: giuseppe@125: In general, the data acquisition unit of a lidar system gets a trigger from the laser to start giuseppe@125: the signal recording. Due to the electronic circuits in the laser and in the data acquisition unit, giuseppe@125: there is always a delay between the outgoing laser pulse and the time at which the acquisition system giuseppe@125: actually starts to record the lidar profile. If this trigger delay is not properly taken into account, giuseppe@125: a systematic error is made in associating each lidar range bin with the corresponding atmospheric range. giuseppe@125: Once the correct measurement of the real trigger delay is done for each detection channel following the giuseppe@125: procedure indicated CARS, such information is inserted in the SCC configuration and HiRELPP correct acquired giuseppe@125: lidar signals for the trigger-delay. giuseppe@125: giuseppe@125: Atmospheric and electronic background subtraction giuseppe@125: ################################################# giuseppe@125: giuseppe@125: The lidar signal has a constant background made of an atmospheric component and an electric component. giuseppe@125: This background can be determined either in the far range of the lidar signal, far enough that the expected giuseppe@125: contribution from atmospheric backscatter is negligible, or in the pre-trigger range before the laser pulse, giuseppe@125: where the signal must be free of electronic distortions. Each one of this option can be defined into HiRELPP. giuseppe@125: Additionally, it is possible to subtract so-called dark signals, which are measured, for example, with a fully giuseppe@125: obscured telescope so that no light from the atmosphere reaches the detectors and only eventual electronic giuseppe@125: distortions are left. This allows HiRELPP to remove potential distortions affecting analog lidar signals. giuseppe@125: giuseppe@125: Low- and high-range automatic signal glueing giuseppe@125: ############################################ giuseppe@125: giuseppe@125: Lidar signals can cover a quite large dynamic range, because the intensity of the light backscattered from the giuseppe@125: aerosol-laden boundary layer in the near range (e.g. at 0.5 km altitude) is several orders of magnitudes higher giuseppe@125: than the intensity of the light backscattered from the rather clean troposphere (e.g. at 10 km altitude). giuseppe@125: As it is demanding to cover this large dynamic range with one data acquisition channel with linear response, giuseppe@125: several approaches are used to overcome this problem. giuseppe@125: One option is to split the signal output from a single photomultiplier into two signals and to record one signal giuseppe@125: using analog detection mode and the other with the photon-counting method. Another option is to split the lidar giuseppe@125: signal optically using a beam splitter and to detect the split components with two detectors and subsequent giuseppe@125: data acquisitions. A third option is to use two (or more) telescopes with separate detection electronics. giuseppe@125: Both SCC preprocessors (HiRELPLP and ELPP) glues the signals for the first 2 options, while gluing is implemented giuseppe@125: directly at optical property level in the third case (ELDA). Before gluing, the near-range and the far-range signals giuseppe@125: need to be screened for low-level clouds, corrected for instrumental effects like dead time, trigger delay, etc., giuseppe@125: and the backgrounds have to be subtracted as explained above. giuseppe@125: HiRELPP and ELPP contains a fully automatic algorithm for the gluing of analog and photon-counting signals as well giuseppe@125: as for the gluing of two photon-counting signals. The algorithm is implemented through three main steps: giuseppe@125: the procedure starts with the determination of a first guess of the gluing region, after that, the algorithm optimizes giuseppe@125: the gluing region performing statistical tests (implemented only in ELPP) and finally, the signals are glued in the giuseppe@125: optimal gluing region. giuseppe@125: giuseppe@125: The typical HiRELPP products are netCDF pre-processed files containing pre-processed (un-calibrated) range corrected giuseppe@125: time series at instrumental vertical and time resolution. If the lidar instrument has polarization capabilities giuseppe@125: the volume linear depolarization ratio is provided as well. giuseppe@125: giuseppe@125: giuseppe@125: CloudScreen giuseppe@125: ------------ giuseppe@125: giuseppe@125: Lidar data contaminated by clouds has to be skipped because the retrieval algorithm implemented in the SCC are optimized giuseppe@125: for aerosol and may produce unreliable results when applied to clouds. giuseppe@125: The aim of the CloudScreen module is to detect clouds by ingesting as input un-calibrated high resolution pre-processed giuseppe@125: range corrected signal timeseries (HiRELPP products). The output of CloudScreen module is a netCDF file containing giuseppe@125: a 2-dimensional grid (x axis: time y axis: altitude) with the same resolution as the corresponding HiRELPP product, giuseppe@125: in which each pixel is flagged as cloud free or cloud contaminated. This information is then transferred to other giuseppe@125: SCC modules for the automatic removal of the cloud contribution within the aerosol optical property products. giuseppe@125: giuseppe@125: ELPP giuseppe@125: ----- giuseppe@125: giuseppe@125: The ELPP module implements all the needed corrections and transformations to be applied to the raw data before they giuseppe@125: can be used to derive the optical products at low temporal/spatial resolution. giuseppe@125: As HiRELPP, ELPP implements correction of some instrumental effects (like for example, dead-time correction, giuseppe@125: trigger-delay correction, overlap correction, atmospheric and electronic background subtraction, giuseppe@125: low- and high-range automatic signal glueing) following the recommendations provided by the EARLINET quality assurance program. giuseppe@125: Additionally, to HiRELPP, time integration or vertical smoothing is performed by ELPP to meet the required condition giuseppe@125: on the products statistical error (defined in the SCC database for each data product type). giuseppe@125: ELPP makes also advantage of the CloudScreen output products so that signals affected by low clouds are automatically removed giuseppe@125: already at level of lidar pre-processor. giuseppe@125: Besides these corrections, ELPP is also responsible to generate the molecular signal needed to calculate giuseppe@125: the aerosol optical products. giuseppe@125: In both aerosol backscatter (Klett, 1981; Fernald, 1984; Di Girolamo et al., 1999; Ansmann et al., 1992a; Ferrare et al., 1998) giuseppe@125: and extinction (Ansmann et al., 1990, 1992b) retrievals the molecular contribution to the atmospheric extinction and giuseppe@125: transmissivity are required as input, which are calculated by ELPP at the emission and detection wavelengths in terms giuseppe@125: of vertical profiles at the same vertical resolution as the pre-processed lidar signals. The molecular number density profile giuseppe@125: is calculated by ELPP from vertical profiles of temperature T(z) and pressure P(z) using the ideal gas law and assuming as 1 giuseppe@125: the value of the air compressibility factor. Temperature and pressure profiles are either calculated from standard atmosphere giuseppe@125: model or taken from the measurements of a close-by radiosounding that can be provided to the SCC giuseppe@125: as a separate input file or provided by model data profiles. giuseppe@125: Once the molecular number density is obtained, the calculation of the molecular optical parameters, i.e., the backscatter giuseppe@125: and extinction coefficients, is done following the procedure reported in Bucholtz (1995) and Miles et al. (2001). giuseppe@125: More details about implemented algorithms in ELPP are reported in D'Amico et al., (2016). giuseppe@125: The typical ELPP products consist of netCDF pre-processed files containing low resolution pre-processed (un-calibrated) giuseppe@125: range corrected time series. giuseppe@125: giuseppe@125: ELDA giuseppe@125: ----- giuseppe@125: giuseppe@125: ELDA applies the algorithms for the retrieval of aerosol optical parameters to the low resolution pre-processed signals, giuseppe@125: produced by ELPP module. The module provides aerosol optical products in a flexible way choosing from a set of giuseppe@125: possible pre-defined analysis procedures. giuseppe@125: giuseppe@125: ELDA implements: giuseppe@125: - retrieval of aerosol extinction profile giuseppe@125: - retrieval of Raman aerosol backscatter profile giuseppe@125: - retrieval of elastic aerosol backscatter profile giuseppe@125: - particle/volume depolarization ratio profile giuseppe@125: giuseppe@125: An automatic vertical-smoothing and time-averaging technique selects the optimal smoothing level as a function of altitude giuseppe@125: on the base of different thresholds on product uncertainties fixed in the SCC database for each product. giuseppe@125: Currently, ELDA delivers only optical products at a single wavelength (so for a multi-wavelength lidar, giuseppe@125: ELDA generates several independent optical products each referring to a single wavelength). giuseppe@125: Full description of implemented algorithms is reported in Mattis et al., (2016). giuseppe@125: For all products and retrieval algorithms, the user can choose whether the statistical uncertainties shall be calculated giuseppe@125: with the Monte Carlo method or by means of error propagation. The only exception are retrievals with the Klett-Fernald a giuseppe@125: lgorithm for which the estimation of uncertainties is implemented only with Monte Carlo method. giuseppe@125: Currently, the separated handling of statistical errors of the lidar signals, of systematic errors of the lidar signals, giuseppe@125: and of uncertainties of the retrieval algorithms is under research within the EARLINET community. giuseppe@125: ELDA allows for the automated vertical smoothing and temporal averaging of the derived products. The user has the giuseppe@125: option to adjust the degree of smoothing and averaging of each individual product by setting several parameters. giuseppe@125: In general, those parameters and constraints can be defined for two different altitude regions, below and above 2 km altitude. giuseppe@125: Two threshold values for the maximum allowable relative statistical error of the product below and above 2 km altitude giuseppe@125: (meaning high expected aerosol load and low aerosol load, respectively) can be defined. giuseppe@125: Beside these user-defined constraints, there are fixed limitations concerning the maximum allowable smoothing and averaging: giuseppe@125: it is not allowed to apply a smoothing that would result in effective vertical resolutions larger than 500m and 2km giuseppe@125: below and above 2km altitude, respectively. giuseppe@125: All methods of calculating profiles of particle backscatter coefficients include a certain calibration procedure. Usually a giuseppe@125: particle-free region in the free troposphere where the aerosol backscatter is assumed as null is used for calibration. giuseppe@125: A calibration window of user-defined width is shifted through the altitude region, where particle-free conditions typically giuseppe@125: occur (user-defined calibration interval). For each window position, the average and standard deviation of the signal or signal giuseppe@125: ratio is calculated. It is assumed that the window position where the signal or signal ratio has its minimum is closest to the giuseppe@125: assumed particle-free conditions. The average value within this calibration window and its standard deviation are used to giuseppe@125: estimate the calibration factor and its statistical uncertainty. If the user knows from ancillary data, e.g., from giuseppe@125: sun-photometer observations or from climatological data of the stratospheric particle load, that there is no giuseppe@125: particle-free altitude layer, it is possible to provide backscatter ratios different from 1 as calibration value. giuseppe@125: ELDA implements the derivative calculation into the aerosol extinction algorithm as derivative of the pre-processed giuseppe@125: signals by weighted or non-weighted linear fit method. giuseppe@125: Finally, concerning the assumptions needed in terms of Angstrom exponent (extinction calculation) and /or lidar ratio giuseppe@125: (elastic backscatter retrieval), it is possible to define in the SCC configuration the values to be used. giuseppe@125: In particular it is possible to include a lidar ratio (Angstrom) profile in order to improve the overall quality of giuseppe@125: the product. These values can be provided to the SCC together with the raw signals and are passed by ELPP to ELDA. giuseppe@125: giuseppe@125: ELIC giuseppe@125: ----- giuseppe@125: giuseppe@125: The ELIC module calibrates both high- and low-resolution pre-processed products (HiRELPP end ELPP products respectively) using the same calibration constant computed by ELDA during the retrieval of low-resolution optical aerosol properties (elastic/Raman bacskcatter calibration). As already mentioned, both HiRELPP and ELPP deliver pre-processed range corrected signal timeseries. Pre-processed range corrected signals are not considered robust lidar products because even if they are proportional to the concentration of atmospheric backscatterers, they depend on specific lidar instrumental characteristics as well. In the retrieval of aerosol optical products (like for example aerosol backscatter), the range corrected signals are used as input and special calibration techniques are used to remove the instrumental dependence. The more is the signal to noise ratio the better is the result of these calibration techniques. Usually, a way to increase the signal to noise ratio is to degrade the time and/or space resolution of the input signals. In general, it is more demanding to get a reliable calibration when working with high resolution lidar data. This is the reason why in the SCC workflow, the calibration is done by ELDA which deals with un-calibrated low-resolution range corrected signals. Anyway, if we assume that the instrumental conditions are stable in the time interval in which the measurements take place, it is possible to use the calibration constants retrieved by ELDA calibrating low resolution signals also to calibrate the high resolution timeseries measured in the same time window. This is the main goal of the ELIC module which runs right after ELDA, gets the calibration constants retrieved by ELDA for all lidar channels and calibrates the corresponding high- and low-resolution range corrected signal timeseries. The ELIC products are netCDF files containing fully calibrated quantities like total attenuated backscatter and volume depolarization ratio. giuseppe@125: giuseppe@125: giuseppe@125: giuseppe@125: ELDEC giuseppe@125: ----- giuseppe@125: giuseppe@125: All the participating stations operate lidar equipped with at least 2 channels detecting independent polarization states of backscattered light and, as consequence, can deliver atmospheric volume/particle depolarization ratio profiles. Anyway, to calculate the volume/particle depolarization ratio from the ratio of these polarization channels an accurate calibration is needed. ELDEC module provide this calibration parameter following the quality assurance procedures defined within ACTRIS CARS (Centre for Aerosol Remote Sensing). In particular, the depolarization calibration is made by submitting to the SCC special raw depolarization calibration datasets. giuseppe@125: giuseppe@125: giuseppe@125: ELQUICK giuseppe@125: ------- giuseppe@125: giuseppe@125: The ELQUICK module generates standardized lidar quicklook for the whole ACTRIS/EARLINET network. Lidar quicklooks (png images) are useful representation of the high resolution timeseries of total attenuated backscatter and/or volume depolarization profiles contained in the ELIC products which can be considered as the two-dimensional pixel grid. The number of vertical pixels of this grid is the number of points of the total attenuated backscatter (or volume depolarization ratio) vertical profile while the number of horizontal pixels is to the number of total attenuated backscatter (or volume depolarization ratio) profiles included in the time series. The color corresponding to each individual pixel is, instead, connected to the value of the total attenuated backscatter (or volume depolarization ratio) at a given altitude and time. In this way, by observing such quicklook images it is easy to visualize aerosol layers and their evolution in both time and space. giuseppe@125: giuseppe@125: giuseppe@125: File Format giuseppe@125: ----------- giuseppe@125: giuseppe@125: All the SCC products are files in Network Common Data Form giuseppe@125: (NetCDF) which is a well known self-describing, machine-independent data giuseppe@125: format that support the creation, access, and sharing of array-oriented giuseppe@125: scientific data. For more information about NetCDF format: giuseppe@125: http://www.unidata.ucar.edu/software/netcdf/. giuseppe@125: giuseppe@125: The NetCDF is a binary format that allows the definition of giuseppe@125: multi-dimensional variables of several types (integers, double, giuseppe@125: character, etc). For each variable it is possible to define one or more giuseppe@125: attributes where to specify variable properties like units, long name, giuseppe@125: description, etc. giuseppe@125: giuseppe@125: It is possible to define global attributes which are not related to a giuseppe@125: specific variable but to the whole file. giuseppe@125: giuseppe@125: A NetCDF file is composed by four different section: giuseppe@125: giuseppe@125: dimensions giuseppe@125: this section contains all the dimensions used in the definition of giuseppe@125: all the variables included in the NetCDF file giuseppe@125: giuseppe@125: variables giuseppe@125: this section contains all the variables stored in the NetCDF file. giuseppe@125: Each variable is defined as a multi-dimensional array of a specific giuseppe@125: type and with all the dimensions defined in the dimensions section giuseppe@125: giuseppe@125: global attributes giuseppe@125: this section lists all the attributes referring to the whole file. As giuseppe@125: the variable the attributes (global or the one attached to a specific giuseppe@125: variable) can be of different type giuseppe@125: giuseppe@125: data giuseppe@125: in this section the data contained in each variable defined in giuseppe@125: variable section is stored. Attribute values (both global or related giuseppe@125: to a specific variable) are not reported in data section but directly giuseppe@125: in variable or global attribute sections. giuseppe@125: giuseppe@125: .. toctree:: giuseppe@125: giuseppe@125: hirelpp_product_format giuseppe@125: cloudscreen_product_format giuseppe@125: elpp_product_format giuseppe@125: eldec_product_format giuseppe@125: elda_product_format giuseppe@125: elic_product_format giuseppe@125: bitmask_flag_description giuseppe@125: giuseppe@125: giuseppe@125: giuseppe@125: