Plugin: node: IBM Aspera High Speed Transfer Server Node🔗
This plugin gives access to capabilities provided by the HSTS Node API.
Authentication uses either a Node API username and password, or an access key and secret, provided with options username and password.
Note
Capabilities of this plugin are used in other plugins that access the Node API, such as aoc, ats, shares.
Note
This plugin can be used with any type of Aspera Node, either on-premises or ATS, provided that you have Node API credentials. Those credentials can be either Node API user or Access Key (for example, on ATS).
File Operations🔗
It is possible to do gen3/node user operations:
browse- Transfer (
upload/download/sync) delete- ...
When using an access key, the so-called gen4/access key API is also supported through subcommands using access_keys do self.
Example:
ascli node browse /: list files with gen3/node user APIascli node access_keys do self browse /: list files with gen4/access key API
Browse🔗
Gen3🔗
This is when executing browse in node plugin or v3.
Native API parameters can be placed in option query.
Special parameters can be placed in option query for "gen3" browse:
| Parameter | Description |
|---|---|
recursive |
Recursively list files. |
max |
Maximum number of files to list. |
self |
Show folder metadata, not content. |
skip |
Single API call. |
Gen4🔗
This is when executing browse in aoc files or in node access_keys do.
Option node_api (Hash) controls some options of API used, with the following parameters:
| Parameter | Default | Description |
|---|---|---|
cache |
true |
true Folder content is retrieved from the Redis database (faster).false Folder content retrieved from storage. |
standard_ports |
true |
true Use hard coded standard ports (33001)false Retrieve server ports from an API call (download_setup) which reads the information from aspera.conf on the server. |
accept_v4 |
true |
true (default) uses API header: Accept-Version: 4.0.false uses legacy method (refer to node API). |
per_page |
- | Requested number of items per API call. Set to nil (@none:) to deactivate paging.Default: 1000 when accept_v4 is true. |
Operation find on gen4/access key🔗
The command find <FOLDER> [filter_expr] is available for gen4/access key, under access_keys do self.
The argument <FOLDER> is mandatory and is the root from which the search is performed.
The argument [filter_expr] is optional and represents the matching criteria.
It recursively scans storage to find files and folders matching the criteria and returns a list of matching entries.
[filter_expr] is either:
- Optional (default): All files and folders are selected
- Type
String: The expression is similar to shell globbing; refer to Ruby function:File.fnmatch - Type
Proc: The expression is a Ruby lambda that takes one argument: aHashthat contains the current folder entry to test. See the following examples.
Examples of expressions:
- Find all files and folders under
/
ascli node access_keys do self find /
- Find all text files in
/Documents
ascli node access_keys do self find /Documents '*.txt'
The following examples show Ruby lambda code you can use in the template command below:
ascli node access_keys do self find / @ruby:'->(f){[code here]}'
Tip
Single quotes are used above to protect the whole Ruby expression from the shell. Then double quotes are used for strings in the Ruby expression to not mix with the shell.
- Find files modified in the last 100 days
->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))<100}
- Find files older than 1 year
->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}
- Find files larger than 1 MB
->(f){f["type"].eql?("file") and f["size"].to_i>1000000}
- Filter out files beginning with
._or named.DS_Store:
->(f){!(f["name"].start_with?("._") or f["name"].eql?(".DS_Store"))}
- Match files using a Ruby Regexp:
\.gif$
->(f){f["name"].match?(/\.gif$/)}
ascli commands can be piped to combine operations, such as find and delete:
ascli node access_keys do self find / @ruby:'->(f){f["type"].eql?("file") and (DateTime.now-DateTime.parse(f["modified_time"]))>365}' --fields=path --format=csv | ascli node --bulk=yes delete @lines:@stdin:
Note
The pipe character | on the last line chains the two commands together.
Listing transfer events🔗
When you run a transfer, its information is stored (typically for 1 day) in the HSTS database (Redis).
To view this information, use the command transfer list.
When you have many transfers, the list is built from multiple API calls.
To list only new events since the last run, use the option once_only:
ascli node transfer list --once-only=yes
ascli stores an iteration_token in its persistence repository to track the latest event.
To reset it, add the option --query=@json:'{"reset": true}'.
To limit how many events are listed, use the max parameter in the query.
Other query parameters are passed through to the underlying API (GET /ops/transfers).
Central🔗
The central sub-command uses the reliable query API (session and file).
Use it to list transfer sessions and transferred files.
To list transferred files:
ascli node central file list
To validate transfers offline, use the validator option.
Note
See the HSTS documentation for more details.
Sync🔗
The node plugin provides three commands for file synchronization:
| Command | node |
shares |
aoc |
server |
Description |
|---|---|---|---|---|---|
sync |
Yes | Yes | Yes | Yes | Perform a local sync by executing async locally. |
async |
Yes | Uses API /async.Get status on sync operations on the server side, as in Aspera Console. |
|||
ssync |
Yes | Uses API /asyncs.Start a sync on the server side and monitor only those operations. |
For details on the sync action, see IBM Aspera Sync.
For the async subcommands show and delete, you can use the special identifier ALL.
FASP Stream🔗
You can start a FASP Stream session from the Node API.
Run the following command:
ascli node stream create @json:<VALUE>
with the following transfer-spec:
{"direction":"send","source":"udp://233.3.3.4:3000?loopback=1&ttl=2","destination":"udp://233.3.3.3:3001/","remote_host":"localhost","remote_user":"stream","remote_password":"<PASSWORD>"}
Watchfolder🔗
For watch folder creation, see Aspera Server documentation or Aspera Watchfolder API Documentation.
You can run watchfolder operations remotely through the Node API:
- Start the
watchdandwatchfolderdservices as a system user that has access to the files - Configure a Watchfolder to define automated transfers
ascli node service create @json:'{"id":"mywatchd","type":"WATCHD","run_as":{"user":"<USERNAME>"}}'
ascli node service create @json:'{"id":"mywatchfolderd","type":"WATCHFOLDERD","run_as":{"user":"<USERNAME>"}}'
ascli node watch_folder create @json:'{"id":"mywfolder","source_dir":"/watch1","target_dir":"/","transport":{"host":"10.25.0.4","user":"<USERNAME>","pass":"<PASSWORD>"}}'
Out of Transfer File Validation🔗
To activate this feature, follow the Aspera Transfer Server configuration.
The following command lists one file that requires validation and assigns it to the unique validator identifier you provide:
ascli node central file list --validator=ascli @json:'{"file_transfer_filter":{"max_result":1}}'
╭──────────────┬──────────────┬────────────┬─────────────────────────────────────╮
│ session_uuid │ file_id │ status │ path │
╞══════════════╪══════════════╪════════════╪═════════════════════════════════════╡
│ 1a74444c-... │ 084fb181-... │ validating │ /home/xfer.../PKG - <TITLE>/200KB.1 │
╰──────────────┴──────────────┴────────────┴─────────────────────────────────────╯
To update the status of the file, use the following command:
ascli node central file modify --validator=ascli @json:'{"files":[{"session_uuid": "1a74444c-...","file_id": "084fb181-...","status": "completed"}]}'
updated
Example: SHOD to ATS🔗
Scenario: Access to a Shares on Demand (SHOD) server on AWS is provided by a partner.
We need to transfer files from this third party SHOD instance into our Azure BLOB storage.
Create an Aspera Transfer Service instance, which provides access to the Node API.
Then create an Option Preset with the Node API URL and credentials of the SHOD instance, named aws_shod.
Create another Option Preset for the Azure ATS instance, named azure_ats.
Then execute the following command:
ascli node download /share/sourcefile --to-folder=/destination_folder --preset=aws_shod --transfer=@preset:azure_ats --transfer.agent=node
This will get transfer information from the SHOD instance and tell the Azure ATS instance to download files.
Node file information🔗
When Node API is used with an Access key, extra information can be retrieved, such as preview.
Note
Display of preview on terminal requires installation of extra gem: rmagick
dnf install -y ImageMagick-devel
gem install rmagick rainbow
For example, it is possible to display the preview of a file, if it exists, using an access key on node:
ascli node access_keys do self thumbnail /preview_samples/Aspera.mpg
Previews are mainly used in AoC; this also works with AoC:
ascli aoc files thumbnail /preview_samples/Aspera.mpg
Tip
To specify the file by its file ID, use the selector syntax: %id:<FILE_ID>
Note
To force textual display of the preview on iTerm, prefix command with: env -u TERM_PROGRAM -u LC_TERMINAL
Creating an access key🔗
ascli node access_keys create @json:'{"id":"<ACCESS_KEY>","secret":"<SECRET>","storage":{"type":"local","path":"/data/mydir"}}'
Tip
The id and secret fields are optional.
If not provided, they will be generated and returned in the result.
In that case, provide option --out.secrets=yes to get the generated secret.
Access keys support extra overriding parameters using parameter: configuration and sub keys transfer and server.
For example, an access key can be modified or created with the following options:
{"configuration":{"transfer":{"target_rate_cap_kbps":500000}}}
The list of supported options can be displayed using command:
ascli node info --fields=@ruby:'/^access_key_configuration_capabilities.*/'
Generating and using a bearer token🔗
Bearer tokens are part of the gen4/access key API.
It follows the model of OAuth 2.
For example, they are used in Aspera on Cloud.
This is also available for developers for any application integrating Aspera.
In this API, files, users and groups are identified by an ID (a String, for example, "125", not necessarily numerical).
Bearer tokens are typically generated by the authenticating application and then recognized by the Node API.
A bearer token is authorized on the node by creating permissions on a folder.
Bearer tokens can be generated using ascli command bearer_token: it takes two arguments:
- The private key used to sign the token.
- The token information, which is a
Hashcontaining the following elements:
| Parameter | Default | Type | Description |
|---|---|---|---|
_scope |
user:all |
Special | Either user:all or admin:all |
_validity |
86400 | Special | Validity in seconds from now. |
user_id |
- | Mandatory | Identifier of user |
scope |
node.<ACCESS_KEY>:_scope |
Mandatory | API scope for example, node.<ACCESS_KEY>:<NODE_SCOPE> |
expires_at |
now+_validity |
Mandatory | Format: %Y-%m-%dT%H:%M:%SZfor example, 2021-12-31T23:59:59Z |
auth_type |
access_key |
Optional | access_key, node_user |
group_ids |
- | Optional | List of group IDs |
organization_id |
- | Optional | Organization ID |
watermarking_json_base64 |
- | Optional | Watermarking information (not used) |
Note
For convenience, ascli provides additional parameters _scope and _validity.
They are not part of the API and are removed from the final payload.
They provide a convenient way to set scope and expires_at.
Bearer token: Environment🔗
An access key shall be created to grant access for transfers to its storage. The access key and its secret represent administrative access to the storage as it has access rights to the whole storage of the access key.
The way to create access keys depends slightly on the type of HSTS:
-
If a self-managed Aspera node is used, then a node user admin must be created: It has no
docrootbut has at least one file restriction (for testing, one can use*to accept creation of an access key with any storage root path). See the Aspera HSTS documentation. -
If Cloud Pak for Integration is used, then the node admin is created automatically.
-
If Aspera on Cloud or ATS is used, then the SaaS API for access key creation is used.
Note
See HSTS manual: Access key authentication section for more details on access key creation.
The following sections assume that an access key has been created and that ascli is configured to use this access key by default using node.
Bearer token: Preparation🔗
Assume that the access key was created, and a default configuration is set to use this access key.
Using ascli, an access key can be created using the access_keys create command on the node (using main node credentials) or on ATS.
Create a private key (organization key) that will be used to sign bearer tokens:
my_private_pem=./myorgkey.pem
ascli config genkey $my_private_pem
Note
This private key is not used for authentication. It is used to sign bearer tokens. See private key for more details on generation.
The corresponding public key shall be placed as an attribute of the access key (done with PUT /access_keys/<ID>):
ascli node access_keys set_bearer_key self @file:$my_private_pem
Note
Either the public or private key can be provided, and only the public key is used. This enables checking the signature of bearer tokens. Above command is executed with access key credentials.
Alternatively, use the following equivalent command, as config genkey also saves the public key with extension .pub:
ascli node access_keys modify %id:self @ruby:'{token_verification_key: File.read("'$my_private_pem'.pub")}'
Bearer token: Configuration for user🔗
- Select a folder for which to grant access to a user, and get its identifier:
my_folder_id=$(ascli node access_keys do self show / --fields=id)
Note
This example uses /, but any folder can be selected in the access key storage.
- Designate a user by ID:
my_user_id=777
Note
This is an arbitrary identifier, typically managed by the web application. Not related to Linux user IDs or anything else.
- Grant this user access to the selected folder:
ascli node access_keys do self permission %id:$my_folder_id create @json:'{"access_type":"user","access_id":"'$my_user_id'"}'
- Create a Bearer token for the user:
ascli node bearer_token @file:./myorgkey.pem @json:'{"user_id":"'$my_user_id'","_validity":3600}' --out.secrets=yes --out.file=bearer.txt
Note
The Bearer token can also be created using command asnodeadmin on HSTS.
See the HSTS manual: Bearer tokens section.
Code for token generation is provided in lib/aspera/api/node.rb
Bearer token: User side🔗
Assume the role of the user, with the following information:
- The URL of the Node API
- A Bearer token
- A file ID for which we have access
To use this information:
ascli node -N --url=https://... --password="Bearer $(cat bearer.txt)" --root-id=$my_folder_id access_keys do self browse /
Tested commands for node🔗
Note
Add ascli node in front of the following commands:
--url=http://localhost:12348 --username=sim --password=<PASSWORD> browse / --fields=path --format=csv --select=@json:'{"basename":"test_file.bin"}'
--url=http://localhost:12348 --username=sim --password=<PASSWORD> transfer list --fields=status
--url=https://node.example.com:9092 --username=user1 --password=<PASSWORD> --ui=text asperabrowser
--url=https://tst.example.com/path --password 'Bearer <ND_BEARER_TOKEN>' --root-id=<ID> access_key do self browse /
access_key create @json:'{"id":"my_username","secret":"<PASSWORD>","storage":{"type":"local","path":"/"}}'
access_key delete my_username
access_key do my_ak_name browse / --secret=<AK_SECRET>
access_key do my_ak_name delete /test_nd_ak2 --secret=<AK_SECRET>
access_key do my_ak_name delete test_nd_ak3 --secret=<AK_SECRET>
access_key do my_ak_name download test_nd_ak3 --to-folder=. --secret=<AK_SECRET>
access_key do my_ak_name find my_test_folder --secret=<AK_SECRET>
access_key do my_ak_name find my_test_folder @re:'\.jpg$' --secret=<AK_SECRET>
access_key do my_ak_name find my_test_folder @ruby:'->(f){f["name"].end_with?(".jpg")}'
--secret=<AK_SECRET>
access_key do my_ak_name mkdir /tst_nd_ak --secret=<AK_SECRET>
access_key do my_ak_name mkfile /mkfile.txt 'hello world' --secret=<AK_SECRET>
access_key do my_ak_name mklink /mklink.txt --query=@json:'{"target":"/mkfile.txt","target_node_id":"123"}' --secret=<AK_SECRET>
access_key do my_ak_name node_info / --secret=<AK_SECRET>
access_key do my_ak_name rename /tst_nd_ak test_nd_ak2 --secret=<AK_SECRET>
access_key do my_ak_name show %id:1 --secret=<AK_SECRET>
access_key do my_ak_name upload 'faux:///test_nd_ak3?100k' --node-api.standard_ports=false --secret=<AK_SECRET>
access_key do self permission %id:<ID> create @json:'{"access_type":"user","access_id":"666"}'
access_key do self permission / delete 1
access_key do self permission / modify 1 @: 'access_levels=@list: read list'
access_key do self permission / show 1
access_key list
access_key set_bearer_key self @file:<PRIVATE_KEY_FILE>
access_key show %id:self
api_details
async bandwidth %name:my_sync_session_name
async counters %name:my_sync_session_name
async delete ALL
async files %name:my_sync_session_name
async files %name:my_sync_session_name --once-only=yes
async list
async show %name:my_sync_session_name
async show ALL
basic_token
browse / --log-level=trace2
cat my_upload_folder/test_file.bin
central file list
central file modify --validator=1 @: files=@list:.
central session list
delete @list:,my_upload_folder/a_folder,my_upload_folder/tdlink,my_upload_folder/a_file
delete my_upload_folder/test_file.bin
download my_upload_folder/test_file.bin --to-folder=.
emulator @json:'{"url":"http://localhost:12348","username":"sim","password":"<PASSWORD>","password_min_bits":0,"docroot":"/data"}'
health
info --fpac='function FindProxyForURL(url,host){return "DIRECT"}'
license
mkdir my_upload_folder/a_folder
mkfile my_upload_folder/a_file1 'hello world'
mklink my_upload_folder/a_folder my_upload_folder/tdlink
rename my_upload_folder a_file1 a_file
search / --query.sort=mtime
service create @json:'{"id":"service1","type":"WATCHD","run_as":{"user":"user1"}}'
service delete service1
service list
slash
space /
spec
ssync bandwidth %name:my_node_sync
ssync counters %name:my_node_sync
ssync create @json:'{"configuration":{"name":"my_node_sync","local":{"path":"my_local_path_real"},"remote":{"host":"my_host","port":my_port,"user":"my_username","pass":"<PASSWORD>","path":"my_remote_path"}}}'
ssync delete %name:my_node_sync
ssync files %name:my_node_sync
ssync list
ssync show %name:my_node_sync
ssync start %name:my_node_sync
ssync state %name:my_node_sync
ssync stop %name:my_node_sync
ssync summary %name:my_node_sync
stream list
sync admin status /data/local_sync
sync pull /aspera-test-dir-tiny --to-folder=/data/local_sync @: reset=true
sync pull /aspera-test-dir-tiny --to-folder=/data/local_sync @json:'{"name":"my_sync_session_name","reset":true}'
transfer bandwidth_average
transfer cancel <ID>
transfer list --query.active_only=true
transfer list --query.reset=true --once-only=yes
transfer modify <ID> @: target_rate_kbps=10000
transfer sessions
transfer show <ID>
transport
upload 'faux:///testfile1?1m' --to-folder=my_local_path
upload --to-folder=my_upload_folder --sources=@ts --ts=@json:'{"paths":[{"source":"/aspera-test-dir-small/10MB.2"}],"precalculate_job_size":true}' --transfer.agent=node --transfer.url=https://node.example.com/path --transfer.username=my_username --transfer.password=<PASSWORD>
upload --username=my_ak_name --password=<AK_SECRET> test_file.bin
upload my_mxf my_docx --ts=@json:'{"target_rate_kbps":1000000,"resume_policy":"none"}'
watch_folder list
Open Telemetry🔗
The node plugin supports Open Telemetry (OTel) for monitoring and tracing.
Note
This is an experimental feature and currently only available for the node plugin and Instana backend.
ascli polls the Node API for transfer events and sends them to an OTel collector.
The command expects the following parameters provided as a Hash positional parameter:
| Parameter | Type | Default | Description |
|---|---|---|---|
url |
String |
- | URL of the Instana HTTPS backend for OTel. |
key |
String |
- | Agent key for the backend. |
interval |
Float |
10 | Polling interval in seconds.0 for single shot. |
To retrieve OTel backend information: Go to the Instana web interface, More → Agents → Docker and identify the agent endpoint and key, for example, endpoint=ingress-blue-saas.instana.io.
Identify the region and the endpoint URL will be https://otlp-[region]-saas.instana.io, that is, replace ingress with otlp.
For convenience, those parameters can be provided in a preset, for example, named otel_default.
ascli config preset initialize otel_default @json:'{"url":"https://otlp-orange-saas.instana.io:4318","key":"<KEY>","interval":1.1}'
Then it is invoked like this (assuming a default node is configured):
ascli node telemetry @preset:otel_default
In Instana, create a custom Dashboard to visualize the OTel data:
- Add Widget: Histogram
- Data Source: Infrastructure and Platforms
- Metric: search
transfer
Node emulator🔗
Note
This is not a feature for production. It's provided for testing only.
The command emulator starts a local web server that answers a subset of the Node API, and executes transfers with the Transfer Daemon (transferd).
It allows testing, without HSTS, the Node API agent (--transfer.agent=node) and the node commands info, browse, transfer list|show|modify|cancel.
The Transfer Daemon and the gem grpc must be installed (see Agent: Transfer Daemon).
The emulator starts its own transferd, and stops it on exit.
It takes an optional Hash argument with the following parameters:
| Field | Type | Description |
|---|---|---|
cert |
String |
Path to the TLS certificate file. Accepted formats: PEM (.pem) or PKCS12 (.p12 / .pfx).Example: /path/to/cert.pem. |
chain |
String |
Path to the PEM certificate chain file (appended as extra chain certificates). Example: /path/to/chain.pem. |
docroot |
String |
Local folder of the node files. Paths of /files/browse and local paths of transfers (sources of send, destination of receive) are relative to it, and confined in it. Defaults to the current working directory.Example: /data/aspera. |
key |
String |
Path to the PEM private key file, or the PKCS12 passphrase when cert is a .p12/.pfx file.Example: /path/to/key.pem. |
password |
String |
Password expected from clients in HTTP Basic authentication, for the above username. Generate it with openssl rand -hex 32: a weak password is refused (see password_min_bits).Example: <PASSWORD>. |
password_min_bits |
Integer |
Minimum estimated strength of password, in bits (log2 of the number of guesses to find it). The strength is estimated with gem zxcvbn if installed, else with a built-in estimate that detects repeats, sequences, keyboard rows, words and years. 0 disables the check. A random password of 32 bytes (openssl rand -hex 32) is far above the default.Default: 80. |
retention_sec |
Integer |
Time in seconds a transfer stays in the list of transfers after it ended (completed, failed or canceled, and no more retried). Default: 86400. |
url |
String |
Address and port the emulator listens on. Use https:// with cert/key for TLS.Default: http://localhost:8080. |
username |
String |
Username expected from clients in HTTP Basic authentication. Set together with password. When not set, requests are accepted without authentication. Mandatory when url is not a loopback address (localhost, 127.0.0.0/8, ::1).Example: node_user. |
For details on url and HTTPS, see Web service.
Like on a real node, the files of the emulator are in its docroot: paths of browse, and local paths of transfers, are relative to it.
Local paths are the sources of an upload (send), and the destination of a download (receive).
Paths leading out of the docroot, also through symbolic links, are rejected.
Start the emulator, it runs until interrupted:
ascli node emulator @json:'{"url":"http://localhost:12348","username":"sim","password":"<PASSWORD>","docroot":"/data"}'
Generate the password with openssl rand -hex 32.
A weak password is refused at startup: its strength is estimated in bits, and shall be at least password_min_bits.
username and password are mandatory when url is not a loopback address (see Web service): anyone reaching the emulator could send the files of the docroot to any host.
Then, in another terminal, use it as a node, for example, to list the files in /data:
ascli node --url=http://localhost:12348 --username=sim --password=<PASSWORD> browse /
Or as the transfer agent, for example, to upload the file /data/my_file.dat to a transfer server:
ascli server upload /my_file.dat --transfer.agent=node --transfer.url=http://localhost:12348 --transfer.username=sim --transfer.password=<PASSWORD>
The transfer is executed by transferd, and its status is available on the emulator:
ascli node --url=http://localhost:12348 --username=sim --password=<PASSWORD> transfer list
Supported endpoints:
| Verb | Path | Action |
|---|---|---|
GET |
/info |
Node information, version and license from transferd. |
GET |
/ops/transfers |
List transfers. Query parameters: active_only, direction, count. |
POST |
/ops/transfers |
Start a transfer with transferd. |
GET |
/ops/transfers/{id} |
Transfer information, with sessions and files. |
PUT |
/ops/transfers/{id} |
Modify target_rate_kbps, min_rate_kbps or rate_policy, or cancel with status: canceled. |
CANCEL |
/ops/transfers/{id} |
Cancel a transfer. |
POST |
/files/browse |
List a folder of the docroot. |
Limitations:
- Only transfers started through the emulator are known. They are kept in memory, and lost when the emulator stops. Ended transfers are removed from the list after
retention_sec(default: one day). - Only Basic authentication is supported: when
usernameandpasswordare set, bearer tokens and access keys are rejected. When they are not set (loopbackurlonly), all requests are accepted. - Not supported:
files/upload_setupandfiles/download_setup(so,node upload|downloadon the emulator),ops/transfers/bandwidth, pause and resume of transfers, query parametersiteration_tokenandtag.