victor@7: Overview i@15: ======== ioannis@0: i@15: This package provides a tool for interacting with the victor@7: Single Calculus Chain through the command line. Specifically, with the script you can: ioannis@0: ioannis@0: * Upload a file to the SCC for processing ioannis@0: * Download the processed files and graphs ioannis@0: * Delete an existing measurement from the SCC (with appropriate privileges) ioannis@0: ioannis@0: ioannis@54: The script provides limited feedback when a file upload fails. Before using ioannis@0: the script, you will need to upload some files manually and be confident that ioannis@0: your SCC file format and processing settings are correct. ioannis@0: i@12: Please note that this is not part of the "official" SCC tools. ioannis@0: ioannis@54: In principle, the main functions are implemented in a class (SCC) that you can also import ioannis@54: and use in your custom scripts. However, error handling has to be improved in this case. If you are interested ioannis@54: in such use, please contact me at ioannis@inoe.ro. ioannis@54: ioannis@0: Any suggestions for improvements and new features are more than welcome. ioannis@0: ioannis@0: Installation ioannis@0: ------------ ioannis@0: victor@9: The easiest way to install this module is from the python package index using pip:: victor@8: ioannis@49: pip install hg+https://repositories.imaa.cnr.it/public/scc_access#egg=scc-access ioannis@0: i@36: or, if you don't have mercurial on your system:: i@36: ioannis@49: pip install https://repositories.imaa.cnr.it/public/scc_access/archive/tip.zip ioannis@0: i@15: You can also use the script by cloning this mercurial repository. ioannis@1: ioannis@0: Settings ioannis@0: -------- ioannis@49: The required user-defined settings need to be specified in a .yaml file. ioannis@0: ioannis@49: The following parameters should be specified:: ioannis@0: ioannis@49: basic_credentials: ['username', 'password'] # The HTTP user name and password that is needed to access the SCC site. ioannis@49: website_credentials: ['username', 'password'] # The user name and password that is needed to log in to the SCC site. ioannis@49: output_dir: /path/to/files/scc_output/ # The directory to download the files ioannis@49: base_url: https://scc.imaa.cnr.it/ # SCC base URL. Normally you shouldn't need to change that. ioannis@0: ioannis@0: ioannis@49: The repository includes a `settings_sample.yaml` file that you can use as a starting point. Rename the file, e.g. to ioannis@49: `settings.yaml` and input the required parameters. If you don't want to specify the file path every time ioannis@49: your run the `scc_access` script, you can name the file `.scc_access.yaml` and place it in your home directory. ioannis@0: ioannis@49: Please note that it's not a good idea to store your stations management credentials in the settings ioannis@49: file. The standard user has "Station Management" privileges and, if the credentials ioannis@0: are stolen, someone could change/delete the stations settings from the SCC database. ioannis@42: For this, it is better to use a user account with minimum access settings, i.e. that ioannis@0: can only upload files and measurements. ioannis@0: ioannis@0: ioannis@0: Usage ioannis@0: ----- ioannis@0: ioannis@0: You can upload a file specifying the username and the system id:: victor@8: ioannis@49: scc_access upload-file 20110101po01.nc 125 ioannis@0: ioannis@0: If you want to wait for the processing to finish and download the resulting files ioannis@49: you can use add the `-p` or `--process` flag:: victor@8: ioannis@49: scc_access upload-file 20110101po01.nc 125 -p ioannis@42: ioannis@49: The two command above assume that you have placed your setting file in the default location. You can specify a ioannis@49: custom location using the -c flag:: ioannis@42: ioannis@49: scc_access -c ./path/to/settings.yaml upload-file 20110101po01.nc 125 ioannis@49: ioannis@54: By default, the SCC will reject an uploaded file if the specified measurement id already exists on the server. You ioannis@49: can instruct the script to delete any existing measurement before uploading using the `--force_upload` flag:: ioannis@0: ioannis@49: scc_access upload-file 20110101po01.nc 125 -p --force_upload ioannis@42: ioannis@54: When uploading a measurement you can require that the processing is delayed by a specified number of hours. This ioannis@54: can be used to assure that your files are processed using model input data, that typically have a delay of more than ioannis@54: 24 hours. You can specify the delay (in hours) using the --delay option:: ioannis@54: ioannis@54: scc_access upload-file 20110101po01.nc 125 --delay 48 ioannis@54: ioannis@54: .. note:: ioannis@54: When specifying a delay, the script will not wait to download the output files. ioannis@54: moritz@29: If you want to delete an existing measurement id from the database use the `delete` moritz@29: command and the measurement id:: victor@8: moritz@29: scc_access delete 20110101po01 ioannis@42: moritz@29: You can list available measurements with the `list` command:: moritz@29: moritz@29: scc_access list ioannis@0: ioannis@49: .. note:: ioannis@49: The `list` command needs to be updated. Cross-check the results before using them. ioannis@0: ioannis@0: For more information on the syntax type:: victor@8: victor@9: scc_access -h ioannis@49: ioannis@49: This will produce the following help text:: ioannis@49: ioannis@49: usage: scc_access [-h] [-d] [-s] [-c CONFIG] ioannis@49: {delete,rerun-all,rerun-elpp,upload-file,list,download} ... ioannis@49: ioannis@49: positional arguments: ioannis@49: {delete,rerun-all,rerun-elpp,upload-file,list,download} ioannis@49: delete Deletes a measurement. ioannis@49: rerun-all Rerun all processing steps for the provided ioannis@49: measurement IDs. ioannis@49: rerun-elpp Rerun low-resolution processing steps for the provided ioannis@49: measurement ID. ioannis@49: upload-file Submit a file and, optionally, download the output ioannis@49: products. ioannis@49: list List measurements registered on the SCC. ioannis@49: download Download selected measurements. ioannis@49: ioannis@49: optional arguments: ioannis@49: -h, --help show this help message and exit ioannis@49: -d, --debug Print debugging information. ioannis@49: -s, --silent Show only warning and error messages. ioannis@49: -c CONFIG, --config CONFIG ioannis@49: Path to the config file. ioannis@49: ioannis@49: You can find out more information about each command e.g. like this:: ioannis@49: ioannis@49: scc_access upload-file -h ioannis@49: ioannis@49: In this case, the help text will give more details about the `upload-file` command:: ioannis@49: ioannis@49: usage: scc_access upload-file [-h] [-p] [--force_upload] ioannis@49: [--radiosounding RADIOSOUNDING] ioannis@49: [--overlap OVERLAP] [--lidarratio LIDARRATIO] ioannis@49: filename system ioannis@49: ioannis@49: positional arguments: ioannis@49: filename Measurement file name or path. ioannis@49: system Processing system id. ioannis@49: ioannis@49: optional arguments: ioannis@49: -h, --help show this help message and exit ioannis@54: --delay DELAY Delay processing by the specified number of hours (0 to 96). ioannis@49: -p, --process Wait for the processing results. ioannis@49: --force_upload If measurement ID exists on SCC, delete before ioannis@49: uploading. ioannis@49: --radiosounding RADIOSOUNDING ioannis@49: Radiosounding file name or path ioannis@49: --overlap OVERLAP Overlap file name or path ioannis@49: --lidarratio LIDARRATIO ioannis@49: Lidar ratio file name or path