docs/data_processing/introduction.rst@5358586fe387 (annotated)
docs/data_processing/introduction.rst
Tue, 05 Apr 2022 12:16:26 +0200
- author
- Giuseppe D'Amico <giuseppe.damico@imaa.cnr.it>
- date
- Tue, 05 Apr 2022 12:16:26 +0200
- changeset 143
- 5358586fe387
- parent 44
- c37ebdf6d03f
- permissions
- -rw-r--r--
Added missing variables in HiRELPP product
| ioannis@37 | 1 | Overview |
| ioannis@37 | 2 | ======== |
| ioannis@37 | 3 | |
| ioannis@37 | 4 | The "Data processing" section of the SCC is used to upload measurements |
| ioannis@37 | 5 | and ancillary files, monitor the processing progress, and view the output |
| ioannis@37 | 6 | results of the SCC. |
| ioannis@37 | 7 | |
| ioannis@42 | 8 | The data processing documentation consist of three subsections: |
| ioannis@37 | 9 | |
| ioannis@37 | 10 | * The procedures of submitting new data for processing are described |
| ioannis@37 | 11 | in the :ref:`uploading_measurements` section. |
| ioannis@37 | 12 | |
| ioannis@37 | 13 | * The procedures of submitting ancillary files (i.e. sounding, overlap, |
| ioannis@37 | 14 | and lidar ratio files) that are used in the processing are described |
| ioannis@37 | 15 | in the :ref:`uploading_ancillary` section. |
| ioannis@37 | 16 | |
| ioannis@37 | 17 | * The procedure to find and explore existing measurements and ancillary |
| ioannis@37 | 18 | files are detailed in the :ref:`viewing_measurements` section. |
| ioannis@37 | 19 | |
| ioannis@37 | 20 | Quick start |
| ioannis@37 | 21 | ----------- |
| ioannis@37 | 22 | |
| ioannis@37 | 23 | .. note:: |
| ioannis@37 | 24 | |
| ioannis@37 | 25 | The following discussion assumed that you have already set-up the system |
| ioannis@37 | 26 | settings in the "Admin" section (see :ref:`administration` for details). |
| ioannis@37 | 27 | |
| ioannis@41 | 28 | Upload a measurement |
| ioannis@41 | 29 | ~~~~~~~~~~~~~~~~~~~~ |
| ioannis@41 | 30 | |
| ioannis@37 | 31 | In order to start processing lidar data, you will need to upload them |
| ioannis@37 | 32 | on the SCC server. The files should be in netcdf format, following the |
| ioannis@37 | 33 | specific format described in :ref:`netcdf_file`. There is no restriction imposed |
| ioannis@37 | 34 | in the filename, but we suggest to use the format <measurement_ID>.nc. You can |
| ioannis@37 | 35 | upload using the "Quick upload" link in the menu. |
| ioannis@37 | 36 | |
| ioannis@37 | 37 | When you submit the processed measurements in the quick upload form, you will |
| ioannis@37 | 38 | need to specify the system configuration that was used to perform the |
| ioannis@37 | 39 | measurement. The *channels* that are defined in the file should be the |
| ioannis@37 | 40 | ones defined in the selected system. The uploading of ancillary files or the |
| ioannis@37 | 41 | selection of categories for the measurement are optional. |
| ioannis@37 | 42 | |
| ioannis@37 | 43 | When the file is uploaded, it is checked if it conforms to the needed |
| ioannis@44 | 44 | netCDF format. The check guarantees that the mandatory variables are present |
| ioannis@37 | 45 | in the file and that they have the correct format. Note that the check is not |
| ioannis@37 | 46 | exhaustive, in the sense it does not guarantee that the file is able to be |
| ioannis@37 | 47 | correctly processed; the aim of this check is to detect as early as possible |
| ioannis@37 | 48 | common problems in the file. If some errors are found, the file will be |
| ioannis@37 | 49 | deleted from the server. |
| ioannis@37 | 50 | |
| ioannis@41 | 51 | |
| ioannis@41 | 52 | Ancillary files |
| ioannis@41 | 53 | ~~~~~~~~~~~~~~~ |
| ioannis@41 | 54 | |
| ioannis@37 | 55 | If ancillary files are needed for the processing two things should be done: |
| ioannis@37 | 56 | |
| ioannis@41 | 57 | #. **The ancillary file names should be defined in the submitted measurement |
| ioannis@41 | 58 | file**. In contrast to the measurement files, the file name of the ancillary |
| ioannis@41 | 59 | files should follow a specified format, described in :ref:`netcdf_file`. In |
| ioannis@41 | 60 | brief, a sounding file should have a name rs_YYYYMMDDccNN.nc, a overlap file |
| ioannis@41 | 61 | a name ov_YYYYMMDDccNN.nc, and a lidar ratio file a name lr_YYYYMMDDccNN.nc, |
| ioannis@44 | 62 | where YYYY is a year, MM is the month, DD is the day, cc is the EARLINET |
| ioannis@41 | 63 | station call sign and NN is a number. |
| ioannis@37 | 64 | |
| ioannis@41 | 65 | #. **The ancillary files should be uploaded on the server**. The uploading can be |
| ioannis@37 | 66 | done in the same form as before, or independently in the "Upload ancillary" |
| ioannis@37 | 67 | form. If an ancillary file required by a measurement is not present in the |
| ioannis@37 | 68 | database, this file will appear as "Missing" and the processing of the |
| ioannis@37 | 69 | measurement will not start and will be marked as "pending". When the missing |
| ioannis@37 | 70 | file is uploaded, its status will change to "OK" and the measurement will be |
| ioannis@37 | 71 | processed. |
| ioannis@41 | 72 | |
| ioannis@41 | 73 | |
| ioannis@41 | 74 | Monitor processing progress |
| ioannis@41 | 75 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| ioannis@41 | 76 | |
| ioannis@41 | 77 | When uploading finishes successfully, you will be transfered to the |
| ioannis@41 | 78 | **measurement status page**. There you can monitor the progress of the |
| ioannis@41 | 79 | processing through the status symbols. The three symbols correspond to the |
| ioannis@41 | 80 | phase of processing: |
| ioannis@41 | 81 | |
| ioannis@41 | 82 | Uploading |
| ioannis@41 | 83 | It indicates if everything needed for processing is provided; practically |
| ioannis@41 | 84 | this indicates if the measurement file and all needed ancillary files have |
| ioannis@41 | 85 | been uploaded. |
| ioannis@41 | 86 | |
| ioannis@41 | 87 | Preprocessing |
| ioannis@41 | 88 | This indicates the progress of the preprocessor module. |
| ioannis@41 | 89 | |
| ioannis@41 | 90 | Optical processing |
| ioannis@41 | 91 | This indicates if progress of the optical processing module (ELDA). |
| ioannis@37 | 92 | |
| ioannis@37 | 93 | |
| ioannis@41 | 94 | These three indicators currently support four states: |
| ioannis@41 | 95 | |
| ioannis@41 | 96 | Gray (Not started) |
| ioannis@41 | 97 | A gray indicator means that the process did not start yet. |
| ioannis@41 | 98 | |
| ioannis@41 | 99 | Orange (In progress) |
| ioannis@41 | 100 | An orange indicator means that the process is currently performed. |
| ioannis@41 | 101 | |
| ioannis@41 | 102 | Green (Success) |
| ioannis@41 | 103 | A green indicator means that the process finished successfully. |
| ioannis@41 | 104 | |
| ioannis@41 | 105 | Red (Fail) |
| ioannis@41 | 106 | A red indicator means that the process failed. |
| ioannis@41 | 107 | |
| ioannis@41 | 108 | |
| ioannis@41 | 109 | While the processing is in progress the page will automatically refresh every |
| ioannis@41 | 110 | 10 seconds. |
| ioannis@41 | 111 | |
| ioannis@41 | 112 | |
| ioannis@41 | 113 | View output |
| ioannis@41 | 114 | ~~~~~~~~~~~ |
| ioannis@41 | 115 | |
| ioannis@41 | 116 | When the processing of your measurement finishes successfully, you can browse |
| ioannis@41 | 117 | and download the results. |
| ioannis@41 | 118 | |
| ioannis@41 | 119 | To **browse** the results, you must select the "Output" tab at the bottom part |
| ioannis@41 | 120 | of the page. There you will find links to view graphs of the resulting optical |
| ioannis@41 | 121 | products. |
| ioannis@41 | 122 | |
| ioannis@41 | 123 | To **download** the results, you can use the links on the "File actions" |
| ioannis@41 | 124 | submenu. Depending on the state of the processing you can downlaod |
| ioannis@41 | 125 | *preprocessed singals*, *optical products*, and *optical product plots*. |
| ioannis@41 | 126 | |
| ioannis@41 | 127 | |
| ioannis@41 | 128 | |
| ioannis@41 | 129 | |
| ioannis@41 | 130 | |
| ioannis@41 | 131 | |
| ioannis@41 | 132 |
