Fri, 21 May 2021 16:26:54 +0300

Fri, 21 May 2021 16:26:54 +0300
changeset 65
parent 63

Initial attempt to list measurements through the API.

victor@7 1 Overview
i@15 2 ========
ioannis@0 3
i@15 4 This package provides a tool for interacting with the
victor@7 5 Single Calculus Chain through the command line. Specifically, with the script you can:
ioannis@0 6
ioannis@0 7 * Upload a file to the SCC for processing
ioannis@0 8 * Download the processed files and graphs
ioannis@0 9 * Delete an existing measurement from the SCC (with appropriate privileges)
ioannis@0 10
ioannis@0 11
ioannis@54 12 The script provides limited feedback when a file upload fails. Before using
ioannis@0 13 the script, you will need to upload some files manually and be confident that
ioannis@0 14 your SCC file format and processing settings are correct.
ioannis@0 15
i@12 16 Please note that this is not part of the "official" SCC tools.
ioannis@0 17
ioannis@54 18 In principle, the main functions are implemented in a class (SCC) that you can also import
ioannis@54 19 and use in your custom scripts. However, error handling has to be improved in this case. If you are interested
ioannis@54 20 in such use, please contact me at
ioannis@54 21
ioannis@0 22 Any suggestions for improvements and new features are more than welcome.
ioannis@0 23
ioannis@0 24 Installation
ioannis@0 25 ------------
ioannis@0 26
victor@9 27 The easiest way to install this module is from the python package index using pip::
ioannis@63 28
ioannis@63 29 pip install scc-access
ioannis@63 30
ioannis@63 31 Alternatively, you can install the latest version directly from the repository::
victor@8 32
ioannis@49 33 pip install hg+
ioannis@0 34
i@36 35 or, if you don't have mercurial on your system::
i@36 36
ioannis@49 37 pip install
ioannis@0 38
i@15 39 You can also use the script by cloning this mercurial repository.
ioannis@1 40
ioannis@0 41 Settings
ioannis@0 42 --------
ioannis@49 43 The required user-defined settings need to be specified in a .yaml file.
ioannis@0 44
ioannis@49 45 The following parameters should be specified::
ioannis@0 46
ioannis@49 47 basic_credentials: ['username', 'password'] # The HTTP user name and password that is needed to access the SCC site.
ioannis@49 48 website_credentials: ['username', 'password'] # The user name and password that is needed to log in to the SCC site.
ioannis@49 49 output_dir: /path/to/files/scc_output/ # The directory to download the files
ioannis@49 50 base_url: # SCC base URL. Normally you shouldn't need to change that.
ioannis@0 51
ioannis@0 52
ioannis@49 53 The repository includes a `settings_sample.yaml` file that you can use as a starting point. Rename the file, e.g. to
ioannis@49 54 `settings.yaml` and input the required parameters. If you don't want to specify the file path every time
ioannis@49 55 your run the `scc_access` script, you can name the file `.scc_access.yaml` and place it in your home directory.
ioannis@0 56
ioannis@49 57 Please note that it's not a good idea to store your stations management credentials in the settings
ioannis@49 58 file. The standard user has "Station Management" privileges and, if the credentials
ioannis@0 59 are stolen, someone could change/delete the stations settings from the SCC database.
ioannis@42 60 For this, it is better to use a user account with minimum access settings, i.e. that
ioannis@0 61 can only upload files and measurements.
ioannis@0 62
ioannis@0 63
ioannis@0 64 Usage
ioannis@0 65 -----
ioannis@0 66
ioannis@0 67 You can upload a file specifying the username and the system id::
victor@8 68
ioannis@49 69 scc_access upload-file 125
ioannis@0 70
ioannis@0 71 If you want to wait for the processing to finish and download the resulting files
ioannis@49 72 you can use add the `-p` or `--process` flag::
victor@8 73
ioannis@49 74 scc_access upload-file 125 -p
ioannis@42 75
ioannis@49 76 The two command above assume that you have placed your setting file in the default location. You can specify a
ioannis@49 77 custom location using the -c flag::
ioannis@42 78
ioannis@49 79 scc_access -c ./path/to/settings.yaml upload-file 125
ioannis@49 80
ioannis@54 81 By default, the SCC will reject an uploaded file if the specified measurement id already exists on the server. You
ioannis@49 82 can instruct the script to delete any existing measurement before uploading using the `--force_upload` flag::
ioannis@0 83
ioannis@49 84 scc_access upload-file 125 -p --force_upload
ioannis@42 85
ioannis@54 86 When uploading a measurement you can require that the processing is delayed by a specified number of hours. This
ioannis@54 87 can be used to assure that your files are processed using model input data, that typically have a delay of more than
ioannis@54 88 24 hours. You can specify the delay (in hours) using the --delay option::
ioannis@54 89
ioannis@54 90 scc_access upload-file 125 --delay 48
ioannis@54 91
ioannis@54 92 .. note::
ioannis@54 93 When specifying a delay, the script will not wait to download the output files.
ioannis@54 94
moritz@29 95 If you want to delete an existing measurement id from the database use the `delete`
moritz@29 96 command and the measurement id::
victor@8 97
moritz@29 98 scc_access delete 20110101po01
ioannis@42 99
moritz@29 100 You can list available measurements with the `list` command::
moritz@29 101
moritz@29 102 scc_access list
ioannis@0 103
ioannis@49 104 .. note::
ioannis@49 105 The `list` command needs to be updated. Cross-check the results before using them.
ioannis@0 106
ioannis@0 107 For more information on the syntax type::
victor@8 108
victor@9 109 scc_access -h
ioannis@49 110
ioannis@49 111 This will produce the following help text::
ioannis@49 112
ioannis@49 113 usage: scc_access [-h] [-d] [-s] [-c CONFIG]
ioannis@49 114 {delete,rerun-all,rerun-elpp,upload-file,list,download} ...
ioannis@49 115
ioannis@49 116 positional arguments:
ioannis@49 117 {delete,rerun-all,rerun-elpp,upload-file,list,download}
ioannis@49 118 delete Deletes a measurement.
ioannis@49 119 rerun-all Rerun all processing steps for the provided
ioannis@49 120 measurement IDs.
ioannis@49 121 rerun-elpp Rerun low-resolution processing steps for the provided
ioannis@49 122 measurement ID.
ioannis@49 123 upload-file Submit a file and, optionally, download the output
ioannis@49 124 products.
ioannis@49 125 list List measurements registered on the SCC.
ioannis@49 126 download Download selected measurements.
ioannis@49 127
ioannis@49 128 optional arguments:
ioannis@49 129 -h, --help show this help message and exit
ioannis@49 130 -d, --debug Print debugging information.
ioannis@49 131 -s, --silent Show only warning and error messages.
ioannis@49 132 -c CONFIG, --config CONFIG
ioannis@49 133 Path to the config file.
ioannis@49 134
ioannis@49 135 You can find out more information about each command e.g. like this::
ioannis@49 136
ioannis@49 137 scc_access upload-file -h
ioannis@49 138
ioannis@49 139 In this case, the help text will give more details about the `upload-file` command::
ioannis@49 140
ioannis@49 141 usage: scc_access upload-file [-h] [-p] [--force_upload]
ioannis@49 142 [--radiosounding RADIOSOUNDING]
ioannis@49 143 [--overlap OVERLAP] [--lidarratio LIDARRATIO]
ioannis@49 144 filename system
ioannis@49 145
ioannis@49 146 positional arguments:
ioannis@49 147 filename Measurement file name or path.
ioannis@49 148 system Processing system id.
ioannis@49 149
ioannis@49 150 optional arguments:
ioannis@49 151 -h, --help show this help message and exit
ioannis@54 152 --delay DELAY Delay processing by the specified number of hours (0 to 96).
ioannis@49 153 -p, --process Wait for the processing results.
ioannis@49 154 --force_upload If measurement ID exists on SCC, delete before
ioannis@49 155 uploading.
ioannis@49 156 --radiosounding RADIOSOUNDING
ioannis@49 157 Radiosounding file name or path
ioannis@49 158 --overlap OVERLAP Overlap file name or path
ioannis@49 159 --lidarratio LIDARRATIO
ioannis@49 160 Lidar ratio file name or path