ioannis@37: Overview ioannis@37: ======== ioannis@37: ioannis@37: The "Data processing" section of the SCC is used to upload measurements ioannis@37: and ancillary files, monitor the processing progress, and view the output ioannis@37: results of the SCC. ioannis@37: ioannis@42: The data processing documentation consist of three subsections: ioannis@37: ioannis@37: * The procedures of submitting new data for processing are described ioannis@37: in the :ref:`uploading_measurements` section. ioannis@37: ioannis@37: * The procedures of submitting ancillary files (i.e. sounding, overlap, ioannis@37: and lidar ratio files) that are used in the processing are described ioannis@37: in the :ref:`uploading_ancillary` section. ioannis@37: ioannis@37: * The procedure to find and explore existing measurements and ancillary ioannis@37: files are detailed in the :ref:`viewing_measurements` section. ioannis@37: ioannis@37: Quick start ioannis@37: ----------- ioannis@37: ioannis@37: .. note:: ioannis@37: ioannis@37: The following discussion assumed that you have already set-up the system ioannis@37: settings in the "Admin" section (see :ref:`administration` for details). ioannis@37: ioannis@41: Upload a measurement ioannis@41: ~~~~~~~~~~~~~~~~~~~~ ioannis@41: ioannis@37: In order to start processing lidar data, you will need to upload them ioannis@37: on the SCC server. The files should be in netcdf format, following the ioannis@37: specific format described in :ref:`netcdf_file`. There is no restriction imposed ioannis@37: in the filename, but we suggest to use the format .nc. You can ioannis@37: upload using the "Quick upload" link in the menu. ioannis@37: ioannis@37: When you submit the processed measurements in the quick upload form, you will ioannis@37: need to specify the system configuration that was used to perform the ioannis@37: measurement. The *channels* that are defined in the file should be the ioannis@37: ones defined in the selected system. The uploading of ancillary files or the ioannis@37: selection of categories for the measurement are optional. ioannis@37: ioannis@37: When the file is uploaded, it is checked if it conforms to the needed ioannis@44: netCDF format. The check guarantees that the mandatory variables are present ioannis@37: in the file and that they have the correct format. Note that the check is not ioannis@37: exhaustive, in the sense it does not guarantee that the file is able to be ioannis@37: correctly processed; the aim of this check is to detect as early as possible ioannis@37: common problems in the file. If some errors are found, the file will be ioannis@37: deleted from the server. ioannis@37: ioannis@41: ioannis@41: Ancillary files ioannis@41: ~~~~~~~~~~~~~~~ ioannis@41: ioannis@37: If ancillary files are needed for the processing two things should be done: ioannis@37: ioannis@41: #. **The ancillary file names should be defined in the submitted measurement ioannis@41: file**. In contrast to the measurement files, the file name of the ancillary ioannis@41: files should follow a specified format, described in :ref:`netcdf_file`. In ioannis@41: brief, a sounding file should have a name rs_YYYYMMDDccNN.nc, a overlap file ioannis@41: a name ov_YYYYMMDDccNN.nc, and a lidar ratio file a name lr_YYYYMMDDccNN.nc, ioannis@44: where YYYY is a year, MM is the month, DD is the day, cc is the EARLINET ioannis@41: station call sign and NN is a number. ioannis@37: ioannis@41: #. **The ancillary files should be uploaded on the server**. The uploading can be ioannis@37: done in the same form as before, or independently in the "Upload ancillary" ioannis@37: form. If an ancillary file required by a measurement is not present in the ioannis@37: database, this file will appear as "Missing" and the processing of the ioannis@37: measurement will not start and will be marked as "pending". When the missing ioannis@37: file is uploaded, its status will change to "OK" and the measurement will be ioannis@37: processed. ioannis@41: ioannis@41: ioannis@41: Monitor processing progress ioannis@41: ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ioannis@41: ioannis@41: When uploading finishes successfully, you will be transfered to the ioannis@41: **measurement status page**. There you can monitor the progress of the ioannis@41: processing through the status symbols. The three symbols correspond to the ioannis@41: phase of processing: ioannis@41: ioannis@41: Uploading ioannis@41: It indicates if everything needed for processing is provided; practically ioannis@41: this indicates if the measurement file and all needed ancillary files have ioannis@41: been uploaded. ioannis@41: ioannis@41: Preprocessing ioannis@41: This indicates the progress of the preprocessor module. ioannis@41: ioannis@41: Optical processing ioannis@41: This indicates if progress of the optical processing module (ELDA). ioannis@37: ioannis@37: ioannis@41: These three indicators currently support four states: ioannis@41: ioannis@41: Gray (Not started) ioannis@41: A gray indicator means that the process did not start yet. ioannis@41: ioannis@41: Orange (In progress) ioannis@41: An orange indicator means that the process is currently performed. ioannis@41: ioannis@41: Green (Success) ioannis@41: A green indicator means that the process finished successfully. ioannis@41: ioannis@41: Red (Fail) ioannis@41: A red indicator means that the process failed. ioannis@41: ioannis@41: ioannis@41: While the processing is in progress the page will automatically refresh every ioannis@41: 10 seconds. ioannis@41: ioannis@41: ioannis@41: View output ioannis@41: ~~~~~~~~~~~ ioannis@41: ioannis@41: When the processing of your measurement finishes successfully, you can browse ioannis@41: and download the results. ioannis@41: ioannis@41: To **browse** the results, you must select the "Output" tab at the bottom part ioannis@41: of the page. There you will find links to view graphs of the resulting optical ioannis@41: products. ioannis@41: ioannis@41: To **download** the results, you can use the links on the "File actions" ioannis@41: submenu. Depending on the state of the processing you can downlaod ioannis@41: *preprocessed singals*, *optical products*, and *optical product plots*. ioannis@41: ioannis@41: ioannis@41: ioannis@41: ioannis@41: ioannis@41: ioannis@41: