SCC Documentation: docs/file_formats/scc_product_format.rst@88d912df1073 (annotated)

docs/file_formats/scc_product_format.rst@88d912df1073 (annotated)

docs/file_formats/scc_product_format.rst

Tue, 27 Apr 2021 12:16:14 +0200

author: Giuseppe D'Amico <giuseppe.damico@imaa.cnr.it>
date: Tue, 27 Apr 2021 12:16:14 +0200
changeset 127: 88d912df1073
parent 126: 2e00e23bad90
permissions: -rw-r--r--

Added FAQ on manual cloud-free

 SCC products
 ============
 Introduction
 ------------
 The Single Calculus Chain (SCC) is the standard EARLINET tool to perform
 automatic and quality checked analysis of raw lidar data. It is
 composed by the following modules:
 -  HiRELPP (High Resolution ELPP)
 -  CloudScreen (SCC cloud screen module)
 -  ELPP (EARLINET Lidar Pre-Processor)
 -  ELDA (EARLINET Lidar Data Analizer)
 -  ELDEC (EARLINET Lidar DEpolarization Calibrator)
 -  ELIC (EARLINET LIdar Calibrator)
 -  ELQUICK (EARLINET Lidar QUICJKlook)
 HiRELPP
 --------
 The HiRELPP module implements the corrections to be applied to the raw lidar signals
 before they can be used to derive higher level products.
 All the operations implemented in HiRELPP are designed to preserve both the vertical and time resolution as high as possible.
 Some instrumental effects (like for example, dead-time correction, trigger-delay correction, overlap correction,
 atmospheric and electronic background subtraction, low- and high-range automatic signal glueing)
 are corrected following the recommendations provided by the EARLINET quality assurance program.
 Dead-time correction
 ####################
 The dead-time corresponds to a maximum count rate. The dead- time causes a non-linearity
 between the actual intensity at the photo-multiplier photocathode and the counted events,
 which can be described theoretically by means of photon statistics. Actual detector can be
 modelled as the paralyzable and the non-paralyzable model. Once information about the model
 to use for describing the counting system and the dead-time value is determined, based on
 standard operating procedures defined by the ACTRIS Center for Aerosol remote Sensing), these
 are provided to HiRELPP and the acquired counts are corrected for the dead-time effect.
 Trigger-delay correction
 ########################
 In general, the data acquisition unit of a lidar system gets a trigger from the laser to start
 the signal recording. Due to the electronic circuits in the laser and in the data acquisition unit,
 there is always a delay between the outgoing laser pulse and the time at which the acquisition system
 actually starts to record the lidar profile. If this trigger delay is not properly taken into account,
 a systematic error is made in associating each lidar range bin with the corresponding atmospheric range.
 Once the correct measurement of the real trigger delay is done for each detection channel following the
 procedure indicated CARS, such information is inserted in the SCC configuration and HiRELPP correct acquired
 lidar signals for the trigger-delay.
 Atmospheric and electronic background subtraction
 #################################################
 The lidar signal has a constant background made of an atmospheric component and an electric component.
 This background can be determined either in the far range of the lidar signal, far enough that the expected
 contribution from atmospheric backscatter is negligible, or in the pre-trigger range before the laser pulse,
 where the signal must be free of electronic distortions. Each one of this option can be defined into HiRELPP.
 Additionally, it is possible to subtract so-called dark signals, which are measured, for example, with a fully
 obscured telescope so that no light from the atmosphere reaches the detectors and only eventual electronic
 distortions are left. This allows HiRELPP to remove potential distortions affecting analog lidar signals.
 Low- and high-range automatic signal glueing
 ############################################
 Lidar signals can cover a quite large dynamic range, because the intensity of the light backscattered from the
 aerosol-laden boundary layer in the near range (e.g. at 0.5 km altitude) is several orders of magnitudes higher
 than the intensity of the light backscattered from the rather clean troposphere (e.g. at 10 km altitude).
 As it is demanding to cover this large dynamic range with one data acquisition channel with linear response,
 several approaches are used to overcome this problem.
 One option is to split the signal output from a single photomultiplier into two signals and to record one signal
 using analog detection mode and the other with the photon-counting method. Another option is to split the lidar
 signal optically using a beam splitter and to detect the split components with two detectors and subsequent
 data acquisitions. A third option is to use two (or more) telescopes with separate detection electronics.
 Both SCC preprocessors (HiRELPLP and ELPP) glues the signals for the first 2 options, while gluing is implemented
 directly at optical property level in the third case (ELDA). Before gluing, the near-range and the far-range signals
 need to be screened for low-level clouds, corrected for instrumental effects like dead time, trigger delay, etc.,
 and the backgrounds have to be subtracted as explained above.
 HiRELPP and ELPP contains a fully automatic algorithm for the gluing of analog and photon-counting signals as well
 as for the gluing of two photon-counting signals. The algorithm is implemented through three main steps:
 the procedure starts with the determination of a first guess of the gluing region, after that, the algorithm optimizes
 the gluing region performing statistical tests (implemented only in ELPP) and finally, the signals are glued in the
 optimal gluing region.
 The typical HiRELPP products are netCDF pre-processed files containing pre-processed (un-calibrated) range corrected
 time series at instrumental vertical and time resolution. If the lidar instrument has polarization capabilities
 the volume linear depolarization ratio is provided as well.
 CloudScreen
 ------------
 Lidar data contaminated by clouds has to be skipped because the retrieval algorithm implemented in the SCC are optimized
 for aerosol and may produce unreliable results when applied to clouds.
 The aim of the CloudScreen module is to detect clouds by ingesting as input un-calibrated high resolution pre-processed
 range corrected signal timeseries (HiRELPP products). The output of CloudScreen module is a netCDF file containing
 a 2-dimensional grid (x axis: time y axis: altitude) with the same resolution as the corresponding HiRELPP product,
 in which each pixel is flagged as cloud free or cloud contaminated. This information is then transferred to other
 SCC modules for the automatic removal of the cloud contribution within the aerosol optical property products.
 ELPP
 -----
 The ELPP module implements all the needed corrections and transformations to be applied to the raw data before they
 can be used to derive the optical products at low temporal/spatial resolution.
 As HiRELPP, ELPP implements correction of some instrumental effects (like for example, dead-time correction,
 trigger-delay correction, overlap correction, atmospheric and electronic background subtraction,
 low- and high-range automatic signal glueing) following the recommendations provided by the EARLINET quality assurance program.
 Additionally, to HiRELPP, time integration or vertical smoothing is performed by ELPP to meet the required condition
 on the products statistical error (defined in the SCC database for each data product type).
 ELPP makes also advantage of the CloudScreen output products so that signals affected by low clouds are automatically removed
 already at level of lidar pre-processor.
 Besides these corrections, ELPP is also responsible to generate the molecular signal needed to calculate
 the aerosol optical products.
 In both aerosol backscatter (Klett, 1981; Fernald, 1984; Di Girolamo et al., 1999; Ansmann et al., 1992a; Ferrare et al., 1998)
 and extinction (Ansmann et al., 1990, 1992b) retrievals the molecular contribution to the atmospheric extinction and
 transmissivity are required as input, which are calculated by ELPP at the emission and detection wavelengths in terms
 of vertical profiles at the same vertical resolution as the pre-processed lidar signals. The molecular number density profile
 is calculated by ELPP from vertical profiles of temperature T(z) and pressure P(z) using the ideal gas law and assuming as 1
 the value of the air compressibility factor. Temperature and pressure profiles are either calculated from standard atmosphere
 model or taken from the measurements of a close-by radiosounding that can be provided to the SCC
 as a separate input file or provided by model data profiles.
 Once the molecular number density is obtained, the calculation of the molecular optical parameters, i.e., the backscatter
 and extinction coefficients, is done following the procedure reported in Bucholtz (1995) and Miles et al. (2001).
 More details about implemented algorithms in ELPP are reported in D'Amico et al., (2016).
 The typical ELPP products consist of netCDF pre-processed files containing low resolution pre-processed (un-calibrated)
 range corrected time series.
 ELDA
 -----
 ELDA applies the algorithms for the retrieval of aerosol optical parameters to the low resolution pre-processed signals,
 produced by ELPP module. The module provides aerosol optical products in a flexible way choosing from a set of
 possible pre-defined analysis procedures.
 ELDA implements:
 - retrieval of aerosol extinction profile
 - retrieval of Raman aerosol backscatter profile
 - retrieval of elastic aerosol backscatter profile
 - particle/volume depolarization ratio profile
 An automatic vertical-smoothing and time-averaging technique selects the optimal smoothing level as a function of altitude
 on the base of different thresholds on product uncertainties fixed in the SCC database for each product.
 Currently, ELDA delivers only optical products at a single wavelength (so for a multi-wavelength lidar,
 ELDA generates several independent optical products each referring to a single wavelength).
 Full description of implemented algorithms is reported in Mattis et al., (2016).
 For all products and retrieval algorithms, the user can choose whether the statistical uncertainties shall be calculated
 with the Monte Carlo method or by means of error propagation. The only exception are retrievals with the Klett-Fernald a
 lgorithm for which the estimation of uncertainties is implemented only with Monte Carlo method.
 Currently, the separated handling of statistical errors of the lidar signals, of systematic errors of the lidar signals,
 and of uncertainties of the retrieval algorithms is under research within the EARLINET community.
 ELDA allows for the automated vertical smoothing and temporal averaging of the derived products. The user has the
 option to adjust the degree of smoothing and averaging of each individual product by setting several parameters.
 In general, those parameters and constraints can be defined for two different altitude regions, below and above 2 km altitude.
 Two threshold values for the maximum allowable relative statistical error of the product below and above 2 km altitude
 (meaning high expected aerosol load and low aerosol load, respectively) can be defined.
 Beside these user-defined constraints, there are fixed limitations concerning the maximum allowable smoothing and averaging:
 it is not allowed to apply a smoothing that would result in effective vertical resolutions larger than 500m and 2km
 below and above 2km altitude, respectively.
 All methods of calculating profiles of particle backscatter coefficients include a certain calibration procedure. Usually a
 particle-free region in the free troposphere where the aerosol backscatter is assumed as null is used for calibration.
 A calibration window of user-defined width is shifted through the altitude region, where particle-free conditions typically
 occur (user-defined calibration interval). For each window position, the average and standard deviation of the signal or signal
 ratio is calculated. It is assumed that the window position where the signal or signal ratio has its minimum is closest to the
 assumed particle-free conditions. The average value within this calibration window and its standard deviation are used to
 estimate the calibration factor and its statistical uncertainty. If the user knows from ancillary data, e.g., from
 sun-photometer observations or from climatological data of the stratospheric particle load, that there is no
 particle-free altitude layer, it is possible to provide backscatter ratios different from 1 as calibration value.
 ELDA implements the derivative calculation into the aerosol extinction algorithm as derivative of the pre-processed
 signals by weighted or non-weighted linear fit method.
 Finally, concerning the assumptions needed in terms of Angstrom exponent (extinction calculation) and /or lidar ratio
 (elastic backscatter retrieval), it is possible to define in the SCC configuration the values to be used.
 In particular it is possible to include a lidar ratio (Angstrom) profile in order to improve the overall quality of
 the product. These values can be provided to the SCC together with the raw signals and are passed by ELPP to ELDA.
 ELIC
 -----
 The ELIC module calibrates both high- and low-resolution pre-processed products (HiRELPP end ELPP products respectively) using the same calibration constant computed by ELDA during the retrieval of low-resolution optical aerosol properties (elastic/Raman bacskcatter calibration). As already mentioned, both HiRELPP and ELPP deliver pre-processed range corrected signal timeseries. Pre-processed range corrected signals are not considered robust lidar products because even if they are proportional to the concentration of atmospheric backscatterers, they depend on specific lidar instrumental characteristics as well. In the retrieval of aerosol optical products (like for example aerosol backscatter), the range corrected signals are used as input and special calibration techniques are used to remove the instrumental dependence. The more is the signal to noise ratio the better is the result of these calibration techniques. Usually, a way to increase the signal to noise ratio is to degrade the time and/or space resolution of the input signals. In general, it is more demanding to get a reliable calibration when working with high resolution lidar data. This is the reason why in the SCC workflow, the calibration is done by ELDA which deals with un-calibrated low-resolution range corrected signals. Anyway, if we assume that the instrumental conditions are stable in the time interval in which the measurements take place, it is possible to use the calibration constants retrieved by ELDA calibrating low resolution signals also to calibrate the high resolution timeseries measured in the same time window. This is the main goal of the ELIC module which runs right after ELDA, gets the calibration constants retrieved by ELDA for all lidar channels and calibrates the corresponding high- and low-resolution range corrected signal timeseries. The ELIC products are netCDF files containing fully calibrated quantities like total attenuated backscatter and volume depolarization ratio.
 ELDEC
 -----
 All the participating stations operate lidar equipped with at least 2 channels detecting independent polarization states of backscattered light and, as consequence, can deliver atmospheric volume/particle depolarization ratio profiles. Anyway, to calculate the volume/particle depolarization ratio from the ratio of these polarization channels an accurate calibration is needed. ELDEC module provide this calibration parameter following the quality assurance procedures defined within ACTRIS CARS (Centre for Aerosol Remote Sensing). In particular, the depolarization calibration is made by submitting to the SCC special raw depolarization calibration datasets.
 ELQUICK
 -------
 The ELQUICK module generates standardized lidar quicklook for the whole ACTRIS/EARLINET network. Lidar quicklooks (png images) are useful representation of the high resolution timeseries of total attenuated backscatter and/or volume depolarization profiles contained in the ELIC products which can be considered as the two-dimensional pixel grid. The number of vertical pixels of this grid is the number of points of the total attenuated backscatter (or volume depolarization ratio) vertical profile while the number of horizontal pixels is to the number of total attenuated backscatter (or volume depolarization ratio) profiles included in the time series. The color corresponding to each individual pixel is, instead, connected to the value of the total attenuated backscatter (or volume depolarization ratio) at a given altitude and time. In this way, by observing such quicklook images it is easy to visualize aerosol layers and their evolution in both time and space.
 File Format
 -----------
 All the SCC products are files in Network Common Data Form
 (NetCDF) which is a well known self-describing, machine-independent data
 format that support the creation, access, and sharing of array-oriented
 scientific data. For more information about NetCDF format:
 http://www.unidata.ucar.edu/software/netcdf/.
 The NetCDF is a binary format that allows the definition of
 multi-dimensional variables of several types (integers, double,
 character, etc). For each variable it is possible to define one or more
 attributes where to specify variable properties like units, long name,
 description, etc.
 It is possible to define global attributes which are not related to a
 specific variable but to the whole file.
 A NetCDF file is composed by four different section:
 dimensions
    this section contains all the dimensions used in the definition of
    all the variables included in the NetCDF file
 variables
    this section contains all the variables stored in the NetCDF file.
    Each variable is defined as a multi-dimensional array of a specific
    type and with all the dimensions defined in the dimensions section
 global attributes
    this section lists all the attributes referring to the whole file. As
    the variable the attributes (global or the one attached to a specific
    variable) can be of different type
 data
    in this section the data contained in each variable defined in
    variable section is stored. Attribute values (both global or related
    to a specific variable) are not reported in data section but directly
    in variable or global attribute sections.
 .. toctree::
  hirelpp_product_format
  cloudscreen_product_format
  elpp_product_format
  eldec_product_format
  elda_product_format
  elic_product_format
  bitmask_flag_description

Mercurial > public > sccdocs / annotate

docs/file_formats/scc_product_format.rst@88d912df1073 (annotated)

docs/file_formats/scc_product_format.rst