Thu, 04 Feb 2021 10:22:30 +0200

Thu, 04 Feb 2021 10:22:30 +0200
changeset 55
parent 54
child 63

Note for future improvements.

victor@7 1 Overview
i@15 2 ========
ioannis@0 3
i@15 4 This package provides a tool for interacting with the
victor@7 5 Single Calculus Chain through the command line. Specifically, with the script you can:
ioannis@0 6
ioannis@0 7 * Upload a file to the SCC for processing
ioannis@0 8 * Download the processed files and graphs
ioannis@0 9 * Delete an existing measurement from the SCC (with appropriate privileges)
ioannis@0 10
ioannis@0 11
ioannis@54 12 The script provides limited feedback when a file upload fails. Before using
ioannis@0 13 the script, you will need to upload some files manually and be confident that
ioannis@0 14 your SCC file format and processing settings are correct.
ioannis@0 15
i@12 16 Please note that this is not part of the "official" SCC tools.
ioannis@0 17
ioannis@54 18 In principle, the main functions are implemented in a class (SCC) that you can also import
ioannis@54 19 and use in your custom scripts. However, error handling has to be improved in this case. If you are interested
ioannis@54 20 in such use, please contact me at
ioannis@54 21
ioannis@0 22 Any suggestions for improvements and new features are more than welcome.
ioannis@0 23
ioannis@0 24 Installation
ioannis@0 25 ------------
ioannis@0 26
victor@9 27 The easiest way to install this module is from the python package index using pip::
victor@8 28
ioannis@49 29 pip install hg+
ioannis@0 30
i@36 31 or, if you don't have mercurial on your system::
i@36 32
ioannis@49 33 pip install
ioannis@0 34
i@15 35 You can also use the script by cloning this mercurial repository.
ioannis@1 36
ioannis@0 37 Settings
ioannis@0 38 --------
ioannis@49 39 The required user-defined settings need to be specified in a .yaml file.
ioannis@0 40
ioannis@49 41 The following parameters should be specified::
ioannis@0 42
ioannis@49 43 basic_credentials: ['username', 'password'] # The HTTP user name and password that is needed to access the SCC site.
ioannis@49 44 website_credentials: ['username', 'password'] # The user name and password that is needed to log in to the SCC site.
ioannis@49 45 output_dir: /path/to/files/scc_output/ # The directory to download the files
ioannis@49 46 base_url: # SCC base URL. Normally you shouldn't need to change that.
ioannis@0 47
ioannis@0 48
ioannis@49 49 The repository includes a `settings_sample.yaml` file that you can use as a starting point. Rename the file, e.g. to
ioannis@49 50 `settings.yaml` and input the required parameters. If you don't want to specify the file path every time
ioannis@49 51 your run the `scc_access` script, you can name the file `.scc_access.yaml` and place it in your home directory.
ioannis@0 52
ioannis@49 53 Please note that it's not a good idea to store your stations management credentials in the settings
ioannis@49 54 file. The standard user has "Station Management" privileges and, if the credentials
ioannis@0 55 are stolen, someone could change/delete the stations settings from the SCC database.
ioannis@42 56 For this, it is better to use a user account with minimum access settings, i.e. that
ioannis@0 57 can only upload files and measurements.
ioannis@0 58
ioannis@0 59
ioannis@0 60 Usage
ioannis@0 61 -----
ioannis@0 62
ioannis@0 63 You can upload a file specifying the username and the system id::
victor@8 64
ioannis@49 65 scc_access upload-file 125
ioannis@0 66
ioannis@0 67 If you want to wait for the processing to finish and download the resulting files
ioannis@49 68 you can use add the `-p` or `--process` flag::
victor@8 69
ioannis@49 70 scc_access upload-file 125 -p
ioannis@42 71
ioannis@49 72 The two command above assume that you have placed your setting file in the default location. You can specify a
ioannis@49 73 custom location using the -c flag::
ioannis@42 74
ioannis@49 75 scc_access -c ./path/to/settings.yaml upload-file 125
ioannis@49 76
ioannis@54 77 By default, the SCC will reject an uploaded file if the specified measurement id already exists on the server. You
ioannis@49 78 can instruct the script to delete any existing measurement before uploading using the `--force_upload` flag::
ioannis@0 79
ioannis@49 80 scc_access upload-file 125 -p --force_upload
ioannis@42 81
ioannis@54 82 When uploading a measurement you can require that the processing is delayed by a specified number of hours. This
ioannis@54 83 can be used to assure that your files are processed using model input data, that typically have a delay of more than
ioannis@54 84 24 hours. You can specify the delay (in hours) using the --delay option::
ioannis@54 85
ioannis@54 86 scc_access upload-file 125 --delay 48
ioannis@54 87
ioannis@54 88 .. note::
ioannis@54 89 When specifying a delay, the script will not wait to download the output files.
ioannis@54 90
moritz@29 91 If you want to delete an existing measurement id from the database use the `delete`
moritz@29 92 command and the measurement id::
victor@8 93
moritz@29 94 scc_access delete 20110101po01
ioannis@42 95
moritz@29 96 You can list available measurements with the `list` command::
moritz@29 97
moritz@29 98 scc_access list
ioannis@0 99
ioannis@49 100 .. note::
ioannis@49 101 The `list` command needs to be updated. Cross-check the results before using them.
ioannis@0 102
ioannis@0 103 For more information on the syntax type::
victor@8 104
victor@9 105 scc_access -h
ioannis@49 106
ioannis@49 107 This will produce the following help text::
ioannis@49 108
ioannis@49 109 usage: scc_access [-h] [-d] [-s] [-c CONFIG]
ioannis@49 110 {delete,rerun-all,rerun-elpp,upload-file,list,download} ...
ioannis@49 111
ioannis@49 112 positional arguments:
ioannis@49 113 {delete,rerun-all,rerun-elpp,upload-file,list,download}
ioannis@49 114 delete Deletes a measurement.
ioannis@49 115 rerun-all Rerun all processing steps for the provided
ioannis@49 116 measurement IDs.
ioannis@49 117 rerun-elpp Rerun low-resolution processing steps for the provided
ioannis@49 118 measurement ID.
ioannis@49 119 upload-file Submit a file and, optionally, download the output
ioannis@49 120 products.
ioannis@49 121 list List measurements registered on the SCC.
ioannis@49 122 download Download selected measurements.
ioannis@49 123
ioannis@49 124 optional arguments:
ioannis@49 125 -h, --help show this help message and exit
ioannis@49 126 -d, --debug Print debugging information.
ioannis@49 127 -s, --silent Show only warning and error messages.
ioannis@49 128 -c CONFIG, --config CONFIG
ioannis@49 129 Path to the config file.
ioannis@49 130
ioannis@49 131 You can find out more information about each command e.g. like this::
ioannis@49 132
ioannis@49 133 scc_access upload-file -h
ioannis@49 134
ioannis@49 135 In this case, the help text will give more details about the `upload-file` command::
ioannis@49 136
ioannis@49 137 usage: scc_access upload-file [-h] [-p] [--force_upload]
ioannis@49 138 [--radiosounding RADIOSOUNDING]
ioannis@49 139 [--overlap OVERLAP] [--lidarratio LIDARRATIO]
ioannis@49 140 filename system
ioannis@49 141
ioannis@49 142 positional arguments:
ioannis@49 143 filename Measurement file name or path.
ioannis@49 144 system Processing system id.
ioannis@49 145
ioannis@49 146 optional arguments:
ioannis@49 147 -h, --help show this help message and exit
ioannis@54 148 --delay DELAY Delay processing by the specified number of hours (0 to 96).
ioannis@49 149 -p, --process Wait for the processing results.
ioannis@49 150 --force_upload If measurement ID exists on SCC, delete before
ioannis@49 151 uploading.
ioannis@49 152 --radiosounding RADIOSOUNDING
ioannis@49 153 Radiosounding file name or path
ioannis@49 154 --overlap OVERLAP Overlap file name or path
ioannis@49 155 --lidarratio LIDARRATIO
ioannis@49 156 Lidar ratio file name or path