comparison: docs/file_formats/scc_product_format.rst
docs/file_formats/scc_product_format.rst
- changeset 125
- 003aa42747f5
- child 126
- 2e00e23bad90
equal
deleted
inserted
replaced
| 124:becb9de86607 | 125:003aa42747f5 |
|---|---|
| 1 SCC products | |
| 2 ============ | |
| 3 | |
| 4 .. warning:: | |
| 5 This section is still under development. | |
| 6 | |
| 7 | |
| 8 Introduction | |
| 9 ------------ | |
| 10 | |
| 11 The Single Calculus Chain (SCC) is the standard EARLINET tool to perform | |
| 12 automatic and quality checked analysis of raw lidar data. It is | |
| 13 composed by the following modules: | |
| 14 | |
| 15 - HiRELPP (High Resolution ELPP) | |
| 16 - CloudScreen | |
| 17 - ELPP (EARLINET Lidar Pre-Processor) | |
| 18 - ELDA (EARLINET Lidar Data Analizer) | |
| 19 - ELDEC (EARLINET Lidar DEpolarization Calibrator) | |
| 20 - ELIC (EARLINET LIdar Calibrator) | |
| 21 - ELQUICK (EARLINET Lidar QUICJKlook) | |
| 22 | |
| 23 | |
| 24 HiRELPP | |
| 25 -------- | |
| 26 | |
| 27 The HiRELPP module implements the corrections to be applied to the raw lidar signals | |
| 28 before they can be used to derive higher level products. | |
| 29 All the operations implemented in HiRELPP are designed to preserve both the vertical and time resolution as high as possible. | |
| 30 Some instrumental effects (like for example, dead-time correction, trigger-delay correction, overlap correction, | |
| 31 atmospheric and electronic background subtraction, low- and high-range automatic signal glueing) | |
| 32 are corrected following the recommendations provided by the EARLINET quality assurance program. | |
| 33 | |
| 34 Dead-time correction | |
| 35 #################### | |
| 36 | |
| 37 The dead-time corresponds to a maximum count rate. The dead- time causes a non-linearity | |
| 38 between the actual intensity at the photo-multiplier photocathode and the counted events, | |
| 39 which can be described theoretically by means of photon statistics. Actual detector can be | |
| 40 modelled as the paralyzable and the non-paralyzable model. Once information about the model | |
| 41 to use for describing the counting system and the dead-time value is determined, based on | |
| 42 standard operating procedures defined by the ACTRIS Center for Aerosol remote Sensing), these | |
| 43 are provided to HiRELPP and the acquired counts are corrected for the dead-time effect. | |
| 44 | |
| 45 | |
| 46 Trigger-delay correction | |
| 47 ######################## | |
| 48 | |
| 49 In general, the data acquisition unit of a lidar system gets a trigger from the laser to start | |
| 50 the signal recording. Due to the electronic circuits in the laser and in the data acquisition unit, | |
| 51 there is always a delay between the outgoing laser pulse and the time at which the acquisition system | |
| 52 actually starts to record the lidar profile. If this trigger delay is not properly taken into account, | |
| 53 a systematic error is made in associating each lidar range bin with the corresponding atmospheric range. | |
| 54 Once the correct measurement of the real trigger delay is done for each detection channel following the | |
| 55 procedure indicated CARS, such information is inserted in the SCC configuration and HiRELPP correct acquired | |
| 56 lidar signals for the trigger-delay. | |
| 57 | |
| 58 Atmospheric and electronic background subtraction | |
| 59 ################################################# | |
| 60 | |
| 61 The lidar signal has a constant background made of an atmospheric component and an electric component. | |
| 62 This background can be determined either in the far range of the lidar signal, far enough that the expected | |
| 63 contribution from atmospheric backscatter is negligible, or in the pre-trigger range before the laser pulse, | |
| 64 where the signal must be free of electronic distortions. Each one of this option can be defined into HiRELPP. | |
| 65 Additionally, it is possible to subtract so-called dark signals, which are measured, for example, with a fully | |
| 66 obscured telescope so that no light from the atmosphere reaches the detectors and only eventual electronic | |
| 67 distortions are left. This allows HiRELPP to remove potential distortions affecting analog lidar signals. | |
| 68 | |
| 69 Low- and high-range automatic signal glueing | |
| 70 ############################################ | |
| 71 | |
| 72 Lidar signals can cover a quite large dynamic range, because the intensity of the light backscattered from the | |
| 73 aerosol-laden boundary layer in the near range (e.g. at 0.5 km altitude) is several orders of magnitudes higher | |
| 74 than the intensity of the light backscattered from the rather clean troposphere (e.g. at 10 km altitude). | |
| 75 As it is demanding to cover this large dynamic range with one data acquisition channel with linear response, | |
| 76 several approaches are used to overcome this problem. | |
| 77 One option is to split the signal output from a single photomultiplier into two signals and to record one signal | |
| 78 using analog detection mode and the other with the photon-counting method. Another option is to split the lidar | |
| 79 signal optically using a beam splitter and to detect the split components with two detectors and subsequent | |
| 80 data acquisitions. A third option is to use two (or more) telescopes with separate detection electronics. | |
| 81 Both SCC preprocessors (HiRELPLP and ELPP) glues the signals for the first 2 options, while gluing is implemented | |
| 82 directly at optical property level in the third case (ELDA). Before gluing, the near-range and the far-range signals | |
| 83 need to be screened for low-level clouds, corrected for instrumental effects like dead time, trigger delay, etc., | |
| 84 and the backgrounds have to be subtracted as explained above. | |
| 85 HiRELPP and ELPP contains a fully automatic algorithm for the gluing of analog and photon-counting signals as well | |
| 86 as for the gluing of two photon-counting signals. The algorithm is implemented through three main steps: | |
| 87 the procedure starts with the determination of a first guess of the gluing region, after that, the algorithm optimizes | |
| 88 the gluing region performing statistical tests (implemented only in ELPP) and finally, the signals are glued in the | |
| 89 optimal gluing region. | |
| 90 | |
| 91 The typical HiRELPP products are netCDF pre-processed files containing pre-processed (un-calibrated) range corrected | |
| 92 time series at instrumental vertical and time resolution. If the lidar instrument has polarization capabilities | |
| 93 the volume linear depolarization ratio is provided as well. | |
| 94 | |
| 95 | |
| 96 CloudScreen | |
| 97 ------------ | |
| 98 | |
| 99 Lidar data contaminated by clouds has to be skipped because the retrieval algorithm implemented in the SCC are optimized | |
| 100 for aerosol and may produce unreliable results when applied to clouds. | |
| 101 The aim of the CloudScreen module is to detect clouds by ingesting as input un-calibrated high resolution pre-processed | |
| 102 range corrected signal timeseries (HiRELPP products). The output of CloudScreen module is a netCDF file containing | |
| 103 a 2-dimensional grid (x axis: time y axis: altitude) with the same resolution as the corresponding HiRELPP product, | |
| 104 in which each pixel is flagged as cloud free or cloud contaminated. This information is then transferred to other | |
| 105 SCC modules for the automatic removal of the cloud contribution within the aerosol optical property products. | |
| 106 | |
| 107 ELPP | |
| 108 ----- | |
| 109 | |
| 110 The ELPP module implements all the needed corrections and transformations to be applied to the raw data before they | |
| 111 can be used to derive the optical products at low temporal/spatial resolution. | |
| 112 As HiRELPP, ELPP implements correction of some instrumental effects (like for example, dead-time correction, | |
| 113 trigger-delay correction, overlap correction, atmospheric and electronic background subtraction, | |
| 114 low- and high-range automatic signal glueing) following the recommendations provided by the EARLINET quality assurance program. | |
| 115 Additionally, to HiRELPP, time integration or vertical smoothing is performed by ELPP to meet the required condition | |
| 116 on the products statistical error (defined in the SCC database for each data product type). | |
| 117 ELPP makes also advantage of the CloudScreen output products so that signals affected by low clouds are automatically removed | |
| 118 already at level of lidar pre-processor. | |
| 119 Besides these corrections, ELPP is also responsible to generate the molecular signal needed to calculate | |
| 120 the aerosol optical products. | |
| 121 In both aerosol backscatter (Klett, 1981; Fernald, 1984; Di Girolamo et al., 1999; Ansmann et al., 1992a; Ferrare et al., 1998) | |
| 122 and extinction (Ansmann et al., 1990, 1992b) retrievals the molecular contribution to the atmospheric extinction and | |
| 123 transmissivity are required as input, which are calculated by ELPP at the emission and detection wavelengths in terms | |
| 124 of vertical profiles at the same vertical resolution as the pre-processed lidar signals. The molecular number density profile | |
| 125 is calculated by ELPP from vertical profiles of temperature T(z) and pressure P(z) using the ideal gas law and assuming as 1 | |
| 126 the value of the air compressibility factor. Temperature and pressure profiles are either calculated from standard atmosphere | |
| 127 model or taken from the measurements of a close-by radiosounding that can be provided to the SCC | |
| 128 as a separate input file or provided by model data profiles. | |
| 129 Once the molecular number density is obtained, the calculation of the molecular optical parameters, i.e., the backscatter | |
| 130 and extinction coefficients, is done following the procedure reported in Bucholtz (1995) and Miles et al. (2001). | |
| 131 More details about implemented algorithms in ELPP are reported in D'Amico et al., (2016). | |
| 132 The typical ELPP products consist of netCDF pre-processed files containing low resolution pre-processed (un-calibrated) | |
| 133 range corrected time series. | |
| 134 | |
| 135 ELDA | |
| 136 ----- | |
| 137 | |
| 138 ELDA applies the algorithms for the retrieval of aerosol optical parameters to the low resolution pre-processed signals, | |
| 139 produced by ELPP module. The module provides aerosol optical products in a flexible way choosing from a set of | |
| 140 possible pre-defined analysis procedures. | |
| 141 | |
| 142 ELDA implements: | |
| 143 - retrieval of aerosol extinction profile | |
| 144 - retrieval of Raman aerosol backscatter profile | |
| 145 - retrieval of elastic aerosol backscatter profile | |
| 146 - particle/volume depolarization ratio profile | |
| 147 | |
| 148 An automatic vertical-smoothing and time-averaging technique selects the optimal smoothing level as a function of altitude | |
| 149 on the base of different thresholds on product uncertainties fixed in the SCC database for each product. | |
| 150 Currently, ELDA delivers only optical products at a single wavelength (so for a multi-wavelength lidar, | |
| 151 ELDA generates several independent optical products each referring to a single wavelength). | |
| 152 Full description of implemented algorithms is reported in Mattis et al., (2016). | |
| 153 For all products and retrieval algorithms, the user can choose whether the statistical uncertainties shall be calculated | |
| 154 with the Monte Carlo method or by means of error propagation. The only exception are retrievals with the Klett-Fernald a | |
| 155 lgorithm for which the estimation of uncertainties is implemented only with Monte Carlo method. | |
| 156 Currently, the separated handling of statistical errors of the lidar signals, of systematic errors of the lidar signals, | |
| 157 and of uncertainties of the retrieval algorithms is under research within the EARLINET community. | |
| 158 ELDA allows for the automated vertical smoothing and temporal averaging of the derived products. The user has the | |
| 159 option to adjust the degree of smoothing and averaging of each individual product by setting several parameters. | |
| 160 In general, those parameters and constraints can be defined for two different altitude regions, below and above 2 km altitude. | |
| 161 Two threshold values for the maximum allowable relative statistical error of the product below and above 2 km altitude | |
| 162 (meaning high expected aerosol load and low aerosol load, respectively) can be defined. | |
| 163 Beside these user-defined constraints, there are fixed limitations concerning the maximum allowable smoothing and averaging: | |
| 164 it is not allowed to apply a smoothing that would result in effective vertical resolutions larger than 500m and 2km | |
| 165 below and above 2km altitude, respectively. | |
| 166 All methods of calculating profiles of particle backscatter coefficients include a certain calibration procedure. Usually a | |
| 167 particle-free region in the free troposphere where the aerosol backscatter is assumed as null is used for calibration. | |
| 168 A calibration window of user-defined width is shifted through the altitude region, where particle-free conditions typically | |
| 169 occur (user-defined calibration interval). For each window position, the average and standard deviation of the signal or signal | |
| 170 ratio is calculated. It is assumed that the window position where the signal or signal ratio has its minimum is closest to the | |
| 171 assumed particle-free conditions. The average value within this calibration window and its standard deviation are used to | |
| 172 estimate the calibration factor and its statistical uncertainty. If the user knows from ancillary data, e.g., from | |
| 173 sun-photometer observations or from climatological data of the stratospheric particle load, that there is no | |
| 174 particle-free altitude layer, it is possible to provide backscatter ratios different from 1 as calibration value. | |
| 175 ELDA implements the derivative calculation into the aerosol extinction algorithm as derivative of the pre-processed | |
| 176 signals by weighted or non-weighted linear fit method. | |
| 177 Finally, concerning the assumptions needed in terms of Angstrom exponent (extinction calculation) and /or lidar ratio | |
| 178 (elastic backscatter retrieval), it is possible to define in the SCC configuration the values to be used. | |
| 179 In particular it is possible to include a lidar ratio (Angstrom) profile in order to improve the overall quality of | |
| 180 the product. These values can be provided to the SCC together with the raw signals and are passed by ELPP to ELDA. | |
| 181 | |
| 182 ELIC | |
| 183 ----- | |
| 184 | |
| 185 The ELIC module calibrates both high- and low-resolution pre-processed products (HiRELPP end ELPP products respectively) using the same calibration constant computed by ELDA during the retrieval of low-resolution optical aerosol properties (elastic/Raman bacskcatter calibration). As already mentioned, both HiRELPP and ELPP deliver pre-processed range corrected signal timeseries. Pre-processed range corrected signals are not considered robust lidar products because even if they are proportional to the concentration of atmospheric backscatterers, they depend on specific lidar instrumental characteristics as well. In the retrieval of aerosol optical products (like for example aerosol backscatter), the range corrected signals are used as input and special calibration techniques are used to remove the instrumental dependence. The more is the signal to noise ratio the better is the result of these calibration techniques. Usually, a way to increase the signal to noise ratio is to degrade the time and/or space resolution of the input signals. In general, it is more demanding to get a reliable calibration when working with high resolution lidar data. This is the reason why in the SCC workflow, the calibration is done by ELDA which deals with un-calibrated low-resolution range corrected signals. Anyway, if we assume that the instrumental conditions are stable in the time interval in which the measurements take place, it is possible to use the calibration constants retrieved by ELDA calibrating low resolution signals also to calibrate the high resolution timeseries measured in the same time window. This is the main goal of the ELIC module which runs right after ELDA, gets the calibration constants retrieved by ELDA for all lidar channels and calibrates the corresponding high- and low-resolution range corrected signal timeseries. The ELIC products are netCDF files containing fully calibrated quantities like total attenuated backscatter and volume depolarization ratio. | |
| 186 | |
| 187 | |
| 188 | |
| 189 ELDEC | |
| 190 ----- | |
| 191 | |
| 192 All the participating stations operate lidar equipped with at least 2 channels detecting independent polarization states of backscattered light and, as consequence, can deliver atmospheric volume/particle depolarization ratio profiles. Anyway, to calculate the volume/particle depolarization ratio from the ratio of these polarization channels an accurate calibration is needed. ELDEC module provide this calibration parameter following the quality assurance procedures defined within ACTRIS CARS (Centre for Aerosol Remote Sensing). In particular, the depolarization calibration is made by submitting to the SCC special raw depolarization calibration datasets. | |
| 193 | |
| 194 | |
| 195 ELQUICK | |
| 196 ------- | |
| 197 | |
| 198 The ELQUICK module generates standardized lidar quicklook for the whole ACTRIS/EARLINET network. Lidar quicklooks (png images) are useful representation of the high resolution timeseries of total attenuated backscatter and/or volume depolarization profiles contained in the ELIC products which can be considered as the two-dimensional pixel grid. The number of vertical pixels of this grid is the number of points of the total attenuated backscatter (or volume depolarization ratio) vertical profile while the number of horizontal pixels is to the number of total attenuated backscatter (or volume depolarization ratio) profiles included in the time series. The color corresponding to each individual pixel is, instead, connected to the value of the total attenuated backscatter (or volume depolarization ratio) at a given altitude and time. In this way, by observing such quicklook images it is easy to visualize aerosol layers and their evolution in both time and space. | |
| 199 | |
| 200 | |
| 201 File Format | |
| 202 ----------- | |
| 203 | |
| 204 All the SCC products are files in Network Common Data Form | |
| 205 (NetCDF) which is a well known self-describing, machine-independent data | |
| 206 format that support the creation, access, and sharing of array-oriented | |
| 207 scientific data. For more information about NetCDF format: | |
| 208 http://www.unidata.ucar.edu/software/netcdf/. | |
| 209 | |
| 210 The NetCDF is a binary format that allows the definition of | |
| 211 multi-dimensional variables of several types (integers, double, | |
| 212 character, etc). For each variable it is possible to define one or more | |
| 213 attributes where to specify variable properties like units, long name, | |
| 214 description, etc. | |
| 215 | |
| 216 It is possible to define global attributes which are not related to a | |
| 217 specific variable but to the whole file. | |
| 218 | |
| 219 A NetCDF file is composed by four different section: | |
| 220 | |
| 221 dimensions | |
| 222 this section contains all the dimensions used in the definition of | |
| 223 all the variables included in the NetCDF file | |
| 224 | |
| 225 variables | |
| 226 this section contains all the variables stored in the NetCDF file. | |
| 227 Each variable is defined as a multi-dimensional array of a specific | |
| 228 type and with all the dimensions defined in the dimensions section | |
| 229 | |
| 230 global attributes | |
| 231 this section lists all the attributes referring to the whole file. As | |
| 232 the variable the attributes (global or the one attached to a specific | |
| 233 variable) can be of different type | |
| 234 | |
| 235 data | |
| 236 in this section the data contained in each variable defined in | |
| 237 variable section is stored. Attribute values (both global or related | |
| 238 to a specific variable) are not reported in data section but directly | |
| 239 in variable or global attribute sections. | |
| 240 | |
| 241 .. toctree:: | |
| 242 | |
| 243 hirelpp_product_format | |
| 244 cloudscreen_product_format | |
| 245 elpp_product_format | |
| 246 eldec_product_format | |
| 247 elda_product_format | |
| 248 elic_product_format | |
| 249 bitmask_flag_description | |
| 250 | |
| 251 | |
| 252 | |
| 253 |
