docs/data_processing/introduction.rst

Wed, 12 Mar 2014 09:58:25 +0200

author
Iannis <ioannis@inoe.ro>
date
Wed, 12 Mar 2014 09:58:25 +0200
changeset 41
c9ba4c828051
parent 37
4e3f530ac84e
child 42
01cfcb387e7f
permissions
-rw-r--r--

Quickstart for measurement processing.

ioannis@37 1 Overview
ioannis@37 2 ========
ioannis@37 3
ioannis@37 4 The "Data processing" section of the SCC is used to upload measurements
ioannis@37 5 and ancillary files, monitor the processing progress, and view the output
ioannis@37 6 results of the SCC.
ioannis@37 7
ioannis@37 8 The data processing section consist of three subsections:
ioannis@37 9
ioannis@37 10 * The procedures of submitting new data for processing are described
ioannis@37 11 in the :ref:`uploading_measurements` section.
ioannis@37 12
ioannis@37 13 * The procedures of submitting ancillary files (i.e. sounding, overlap,
ioannis@37 14 and lidar ratio files) that are used in the processing are described
ioannis@37 15 in the :ref:`uploading_ancillary` section.
ioannis@37 16
ioannis@37 17 * The procedure to find and explore existing measurements and ancillary
ioannis@37 18 files are detailed in the :ref:`viewing_measurements` section.
ioannis@37 19
ioannis@37 20 Quick start
ioannis@37 21 -----------
ioannis@37 22
ioannis@37 23 .. note::
ioannis@37 24
ioannis@37 25 The following discussion assumed that you have already set-up the system
ioannis@37 26 settings in the "Admin" section (see :ref:`administration` for details).
ioannis@37 27
ioannis@41 28 Upload a measurement
ioannis@41 29 ~~~~~~~~~~~~~~~~~~~~
ioannis@41 30
ioannis@37 31 In order to start processing lidar data, you will need to upload them
ioannis@37 32 on the SCC server. The files should be in netcdf format, following the
ioannis@37 33 specific format described in :ref:`netcdf_file`. There is no restriction imposed
ioannis@37 34 in the filename, but we suggest to use the format <measurement_ID>.nc. You can
ioannis@37 35 upload using the "Quick upload" link in the menu.
ioannis@37 36
ioannis@37 37 When you submit the processed measurements in the quick upload form, you will
ioannis@37 38 need to specify the system configuration that was used to perform the
ioannis@37 39 measurement. The *channels* that are defined in the file should be the
ioannis@37 40 ones defined in the selected system. The uploading of ancillary files or the
ioannis@37 41 selection of categories for the measurement are optional.
ioannis@37 42
ioannis@37 43 When the file is uploaded, it is checked if it conforms to the needed
ioannis@37 44 netcdf format. The check guarantees that the mandatory variables are present
ioannis@37 45 in the file and that they have the correct format. Note that the check is not
ioannis@37 46 exhaustive, in the sense it does not guarantee that the file is able to be
ioannis@37 47 correctly processed; the aim of this check is to detect as early as possible
ioannis@37 48 common problems in the file. If some errors are found, the file will be
ioannis@37 49 deleted from the server.
ioannis@37 50
ioannis@41 51
ioannis@41 52 Ancillary files
ioannis@41 53 ~~~~~~~~~~~~~~~
ioannis@41 54
ioannis@37 55 If ancillary files are needed for the processing two things should be done:
ioannis@37 56
ioannis@41 57 #. **The ancillary file names should be defined in the submitted measurement
ioannis@41 58 file**. In contrast to the measurement files, the file name of the ancillary
ioannis@41 59 files should follow a specified format, described in :ref:`netcdf_file`. In
ioannis@41 60 brief, a sounding file should have a name rs_YYYYMMDDccNN.nc, a overlap file
ioannis@41 61 a name ov_YYYYMMDDccNN.nc, and a lidar ratio file a name lr_YYYYMMDDccNN.nc,
ioannis@41 62 where YYYY is a year, MM is the month, DD is the day, cc is the earlinet
ioannis@41 63 station call sign and NN is a number.
ioannis@37 64
ioannis@41 65 #. **The ancillary files should be uploaded on the server**. The uploading can be
ioannis@37 66 done in the same form as before, or independently in the "Upload ancillary"
ioannis@37 67 form. If an ancillary file required by a measurement is not present in the
ioannis@37 68 database, this file will appear as "Missing" and the processing of the
ioannis@37 69 measurement will not start and will be marked as "pending". When the missing
ioannis@37 70 file is uploaded, its status will change to "OK" and the measurement will be
ioannis@37 71 processed.
ioannis@41 72
ioannis@41 73
ioannis@41 74 Monitor processing progress
ioannis@41 75 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
ioannis@41 76
ioannis@41 77 When uploading finishes successfully, you will be transfered to the
ioannis@41 78 **measurement status page**. There you can monitor the progress of the
ioannis@41 79 processing through the status symbols. The three symbols correspond to the
ioannis@41 80 phase of processing:
ioannis@41 81
ioannis@41 82 Uploading
ioannis@41 83 It indicates if everything needed for processing is provided; practically
ioannis@41 84 this indicates if the measurement file and all needed ancillary files have
ioannis@41 85 been uploaded.
ioannis@41 86
ioannis@41 87 Preprocessing
ioannis@41 88 This indicates the progress of the preprocessor module.
ioannis@41 89
ioannis@41 90 Optical processing
ioannis@41 91 This indicates if progress of the optical processing module (ELDA).
ioannis@37 92
ioannis@37 93
ioannis@41 94 These three indicators currently support four states:
ioannis@41 95
ioannis@41 96 Gray (Not started)
ioannis@41 97 A gray indicator means that the process did not start yet.
ioannis@41 98
ioannis@41 99 Orange (In progress)
ioannis@41 100 An orange indicator means that the process is currently performed.
ioannis@41 101
ioannis@41 102 Green (Success)
ioannis@41 103 A green indicator means that the process finished successfully.
ioannis@41 104
ioannis@41 105 Red (Fail)
ioannis@41 106 A red indicator means that the process failed.
ioannis@41 107
ioannis@41 108
ioannis@41 109 While the processing is in progress the page will automatically refresh every
ioannis@41 110 10 seconds.
ioannis@41 111
ioannis@41 112
ioannis@41 113 View output
ioannis@41 114 ~~~~~~~~~~~
ioannis@41 115
ioannis@41 116 When the processing of your measurement finishes successfully, you can browse
ioannis@41 117 and download the results.
ioannis@41 118
ioannis@41 119 To **browse** the results, you must select the "Output" tab at the bottom part
ioannis@41 120 of the page. There you will find links to view graphs of the resulting optical
ioannis@41 121 products.
ioannis@41 122
ioannis@41 123 To **download** the results, you can use the links on the "File actions"
ioannis@41 124 submenu. Depending on the state of the processing you can downlaod
ioannis@41 125 *preprocessed singals*, *optical products*, and *optical product plots*.
ioannis@41 126
ioannis@41 127
ioannis@41 128
ioannis@41 129
ioannis@41 130
ioannis@41 131
ioannis@41 132

mercurial