Mercurial > public > sccdocs / file revision

Overview
========

The "Data processing" section of the SCC is used to upload measurements 
and ancillary files, monitor the processing progress, and view the output 
results of the SCC.

The data processing documentation consist of three subsections:

* The procedures of submitting new data for processing are described 
  in the :ref:`uploading_measurements` section.

* The procedures of submitting ancillary files (i.e. sounding, overlap, 
  and lidar ratio files) that are used in the processing are described 
  in the :ref:`uploading_ancillary` section.
  
* The procedure to find and explore existing measurements and ancillary 
  files are detailed in the :ref:`viewing_measurements` section.

Quick start
-----------

.. note::

   The following discussion assumed that you have already set-up the system 
   settings in the "Admin" section (see :ref:`administration` for details).

Upload a measurement
~~~~~~~~~~~~~~~~~~~~

In order to start processing lidar data, you will need to upload them 
on the SCC server. The files should be in netcdf format, following the 
specific format described in :ref:`netcdf_file`. There is no restriction imposed
in the filename, but we suggest to use the format <measurement_ID>.nc. You can 
upload using the "Quick upload" link in the menu.

When you submit the processed measurements in the quick upload form, you will 
need to specify the system configuration that was used to perform the 
measurement. The *channels* that are defined in the file should be the
ones defined in the selected system. The uploading of ancillary files or the
selection of categories for the measurement are optional.

When the file is uploaded, it is checked if it conforms to the needed
netCDF format. The check guarantees that the mandatory variables are present
in the file and that they have the correct format. Note that the check is not 
exhaustive, in the sense it does not guarantee that the file is able to be 
correctly processed; the aim of this check is to detect as early as possible
common problems in the file. If some errors are found, the file will be 
deleted from the server.


Ancillary files
~~~~~~~~~~~~~~~

If ancillary files are needed for the processing two things should be done:

#. **The ancillary file names should be defined in the submitted measurement 
   file**. In contrast to the measurement files, the file name of the ancillary 
   files should follow a specified format, described in :ref:`netcdf_file`. In 
   brief, a sounding file should have a name rs_YYYYMMDDccNN.nc, a overlap file 
   a name ov_YYYYMMDDccNN.nc, and a lidar ratio file a name lr_YYYYMMDDccNN.nc, 
   where YYYY is a year, MM is the month, DD is the day, cc is the EARLINET 
   station call sign and NN is a number.

#. **The ancillary files should be uploaded on the server**. The uploading can be
   done in the same form as before, or independently in the "Upload ancillary"
   form. If an ancillary file required by a measurement is not present in the
   database, this file will appear as "Missing" and the processing of the 
   measurement will not start and will be marked as "pending". When the missing 
   file is uploaded, its status will change to "OK" and the measurement will be 
   processed.


Monitor processing progress
~~~~~~~~~~~~~~~~~~~~~~~~~~~

When uploading finishes successfully, you will be transfered to the 
**measurement status page**. There you can monitor the progress of the 
processing through the status symbols. The three symbols correspond to the 
phase of processing:

Uploading
   It indicates if everything needed for processing is provided; practically
   this indicates if  the measurement file and all needed ancillary files have 
   been uploaded.

Preprocessing
    This indicates the progress of the preprocessor module.
    
Optical processing
    This indicates if progress of the optical processing module (ELDA).


These three indicators currently support four states:

Gray (Not started)
    A gray indicator means that the process did not start yet.

Orange (In progress)
    An orange indicator means that the process is currently performed.

Green (Success)
    A green indicator means that the process finished successfully.

Red (Fail)
    A red indicator means that the process failed.


While the processing is in progress the page will automatically refresh every
10 seconds.


View output
~~~~~~~~~~~

When the processing of your measurement finishes successfully, you can browse 
and download the results.

To **browse** the results, you must select the "Output" tab at the bottom part 
of the page. There you will find links to view graphs of the resulting optical
products.

To **download** the results, you can use the links on the "File actions" 
submenu. Depending on the state of the processing you can downlaod 
*preprocessed singals*, *optical products*, and *optical product plots*.