Operational Utilities🔗
This section covers the specialized modules and utilities used to integrate ascli into your broader operational infrastructure.
While the core plugins handle data movement, these tools provide the "integration layer" for enterprise environments: Aspera Sync and Hot Folder enable automated, folder-based synchronization; Nagios and SMTP modules provide health monitoring and automated email alerting for transfer status; and the asession tool and the Ruby module Aspera allow integration of transfers into other programs.
Together, these features transform the CLI from a manual tool into a fully integrated component of an automated, monitored data workflow.
IBM Aspera Sync🔗
An interface for the async utility is provided in the following plugins:
server sync(SSH Auth)node sync(use gen3 token)aoc files sync(usesnodeplugin with bearer token)shares files sync(usesnodeplugin with gen3 token)
The sync command, available in above plugins, performs the following actions:
- Start a local Sync session by executing the
asynccommand with the appropriate parameters. - Get local Sync session information by accessing the Async snap database directly.
- Get local Sync session information using the
asyncadmincommand, if available.
One advantage of using ascli over the async command line is the possibility to use a configuration file, using standard options of ascli.
Moreover, ascli supports sync with applications requiring token-based authorization.
Some sync parameters are filled by the related plugin using transfer spec parameters (for example, including token).
Note
All sync commands require an async enabled license and availability of the async executable (and asyncadmin).
The Aspera Transfer Daemon 1.3+ includes this.
Quick test🔗
Below is a simple end-to-end procedure to verify synchronization using the demo server.
-
Initialize the demo server configuration:
ascli config initdemo -
Create a local folder with a sample file to sync:
mkdir foobar echo hello > foobar/file1 -
Create the destination folder on the remote server:
ascli server -Pdemoserver mkdir /Upload/mydest1 -
Run the sync operation:
ascli server -Pdemoserver sync push foobar --to-folder=/Upload/mydest1
The following sections provide additional details on available options and configuration.
Starting a sync session🔗
To start a sync session, use one of the three sync directions, followed by a folder path.
For push and bidi, the path is a local folder.
For pull, the path is a remote folder.
The corresponding path on the opposite side is provided using the to_folder option.
General syntax:
ascli ... sync <DIRECTION> <PATH> [--to-folder=<PATH>] [<SYNC_INFO>]
(parameter) |
(parameter) |
to_folder(option) |
|---|---|---|
push |
Local | Remote |
bidi |
Local | Remote |
pull |
Remote | Local |
An optional positional Hash argument (<SYNC_INFO>) may be provided.
It can be expressed in one of two formats:
confformat (recommended)argsformat (legacy)
A single sync session must use one format exclusively.
-
If
<SYNC_INFO>is not provided, a default configuration is automatically generated in theconfformat (details in the next section). -
If argument
<SYNC_INFO>is provided, the format is inferred: -
If the
Hashcontains either keysessionsorinstance, theargsformat is used. - Otherwise, the
confformat is used.
sync_info: conf format🔗
This is the preferred syntax.
It is the same payload as specified on the async option --conf or in Node API /asyncs.
Documentation on Async Node API can be found on IBM Developer Portal.
The following parameters are automatically filled from mandatory arguments, and are not allowed:
directionlocal.pathremote.path
Parameter name is set to a default value if not provided in sync_info.
Parameter quiet is set to false if not provided in sync_info and a terminal is detected.
The documentation is available in the terminal with:
ascli config sync spec
Note
ascli accepts the following fields within the sync_info Hash.
The options listed in the Description column correspond to the equivalent parameters used by the low-level async command.
| Field | Type | Description |
|---|---|---|
ascp_dir |
String |
Directory containing ascp executable to use. |
assume_no_mods |
Bool |
Assume that the directory structure has not been modified. Default: false.( --assume-no-mods) |
checksum |
Enum[String] |
Use the specified checksum type. Default is none on cloud storage. Allowed values: sha1, md5, sha1_sparse, md5_sparse, none.Default: sha1_sparse.( --checksum={enum})(-k) |
clean_excluded |
Bool |
Removes any existing entries in the snapshot database for excluded paths Default: false.( --clean-excluded) |
cookie |
String |
User-defined identification string. ( --cookie={string}) |
cooloff_max_seconds |
Integer |
Wait up to the specified time for a file to stop changing before skipping synchronization of the file. 0 for disabled Default: 0.( --cooloff-max={integer}) |
cooloff_seconds |
Integer |
Delay the start of the transfer to confirm that the content is not changing. Value must be between 0 and 60 Default: 3.( --cooloff={integer}) |
create_dir |
Bool |
Create the source directory, target directory, or both, if they do not exist. Default: false.( --create-dir) |
db_cache_size |
Integer |
Specify DB cache size. Default: 16000. |
db_journal_off |
Bool |
Turn off DB journal. Default: false. |
db_sync_on |
Bool |
Enable synchronous write in DB. Default: false. |
dedup |
Enum[String] |
Take the specified action when async detects duplicate files on the source. Allowed values: copy, inode, hardlink, none.Default: none.( --dedup={enum}) |
delete_before |
Bool |
Schedule deletes before transfers. Default: false.( --delete-before) |
delete_delay |
Bool |
Delay actual deletes until the end of the synchronization. Default: false.( --delete-delay) |
direction |
Enum[String] |
The direction of replication relative to the local. Allowed values: bidi, pull, push.Default: push.( --direction={enum})(-K) |
exclude_dirs_older_than |
Hash |
Don't scan directories with a recursive modified time older than absolute or async start time - relative_seconds |
exclude_dirs_older_than.absolute |
String |
UTC timestamp. Empty value for disabled. |
exclude_dirs_older_than.relative_seconds |
Integer |
Relative to async start time. -1 for disabled.Default: -1. |
filters |
Array[Hash] |
The filters allow to further specify which files have to be excluded and included from the transfer list. Each filter is defined by a rule and a value. Order of filters matters |
filters[].rule |
Enum[String] |
The rule for the filter. Required. Allowed values: include, exclude, include_from, exclude_from. |
filters[].value |
String |
On include or exclude, the filter's pattern. On include_from or exclude_from, the path containing filter specifications Required. |
ignore_delete |
Bool |
Do not copy removals to the peer. Default: false.( --ignore-delete) |
ignore_mode |
Bool |
Source files that have had their mode changed after the initial transfer will not update the destination file mode. Default: false.( --ignore-mode) |
ignore_remote_host_sync_name |
Bool |
Do not check that the remote host being used for the current transfer matches the host used when the local database was created Default: false. |
local |
Hash |
Required. |
local.pass |
String |
Authenticate the local async with the specified password. |
local.path |
String |
The directory to be synchronized on the local host. Required. ( --local-dir={string})(-d) |
local_apply_docroot |
Bool |
Prepend the docroot to the directory on the local host. Default: false.( --apply-local-docroot) |
local_checksum_threads |
Integer |
Maximum number of threads to do checksum on the local host. Value must be between 1 and 99. Default: 4.( --local-checksum-threads={integer}) |
local_db_dir |
String |
Use the specified database directory on the local host. Default is .private-asp at the root level of the synchronized directory.( --local-db-dir={string})(-b) |
local_db_store_dir |
String |
Store/Restore the database to/from the specified directory on the local host. The value can be an absolute path, an URI or - (use the local sync dir) ( --local-db-store-dir={string}) |
local_force_stat |
Bool |
Forces the local async to retrieve file information even when no changes are detected by the scanner or monitor. Default: false.( --local-force-stat) |
local_fs_threads |
Integer |
Maximum number of threads to do file system operations on the local host. Value must be between 1 and 99. Default: 1.( --local-fs-threads={integer}) |
local_keep_dir |
String |
Move deleted files into the specified directory on the local host. ( --keep-dir-local={string}) |
local_mount_signature |
String |
Verify that the file system is mounted by the existence of this file on the local host. ( --local-mount-signature={string}) |
local_move_cache_timeout_seconds |
Integer |
Delay in seconds before aborting moving a file from local cache to final destination. -1 for disabled.Default: -1.( --local-move-cache-timeout={integer}) |
local_preserve_acls |
Enum[String] |
Preserve access control lists on the local host. Allowed values: native, metafile, none.Default: none.( --preserve-acls={enum}) |
local_preserve_xattrs |
Enum[String] |
Preserve extended attributes on the local. Allowed values: native, metafile, none.Default: none.( --preserve-xattrs={enum}) |
local_scan_interval_milliseconds |
Integer |
Enable periodic scans on the local host during a continuous sync. -1 for disabled. On the command line, the value is a duration, e.g. 1h 30m, 2s500ms, 5:30, 1d 02:00:00.250. A plain number is in seconds. Stored in milliseconds.Default: -1.( --scan-interval={integer}) |
local_scan_threads |
Integer |
Number of directory scanning threads on the local host. Value must be between 1 and 99 Default: 1.( --scan-threads={integer}) |
local_stat_cache_size |
Integer |
Set stat cache size on the local host. 0 for disabled. Default: 0. |
log |
Hash |
|
log.level |
Enum[String] |
Use the specified log level. Allowed values: log, dbg1, dbg2.Default: log.(special: -D) |
log.local_dir |
String |
Use the specified logging directory on the local host. ( --alt-logdir={string})(-L) |
log.remote_dir |
String |
Use the specified logging directory on the remote host. ( --remote-logdir={string})(-R) |
manifest_path |
String |
A directory path where ascp will create manifest TEXT files (passed to ascp as --file-manifest-path) |
mirror |
Bool |
Force the pulling side to be exactly like the pushing side, removing files on the destination that don't exist on the source and resending source files that don't have an exact match on the destination. Cannot be used in bi-directional mode. Default: false.( --mirror) |
mode |
Enum[String] |
Specify whether async runs continuously or not. In one_time mode, async stops after the first full synchronization. continuous supported only if the source is Windows or Linux.Allowed values: one_time, continuous.Default: one_time.(special: --continuous)(-C) |
monitor_buffer_size |
Integer |
Bytes to allocate for the change monitor buffer. Applies to any Windows machine on either side. -1 to use the computed value.Default: -1. |
name |
String |
Name of the synchronization pair. Required. ( --name={string})(-N) |
no_log |
String |
Suppress log messages for ITEM. The only currently supported ITEM is 'stats', which suppresses both STATS and PROG log messages. ( --no-log={string}) |
no_preserve_root_attrs |
Bool |
Disable the preservation of attributes on the Sync root. Default: false.( --no-preserve-root-attrs) |
no_scan |
Bool |
Skip initial scanning. Default: false.( --no-scan) |
notifications_sharing_retry_max |
Integer |
Retry processing filesystem notifications up to the specified maximum number after a sharing violation. Default: 3. |
overwrite |
Enum[String] |
Overwrite files according to the specified policy. Default is determined by the direction: conflict for bidi, otherwise always.Allowed values: always, older, conflict.( --overwrite={enum})(-o) |
pending_max |
Integer |
Allow the maximum number of files that are pending transfer to be no more than the specified number. Default: 2000.( --pending-max={integer}) |
preserve_access_time |
Bool |
Preserve file access time from the source to the destination. Default: false.( --preserve-access-time) |
preserve_creation_time |
Bool |
Preserve file creation time from the source to the destination. Default: false.( --preserve-creation-time) |
preserve_gid |
Bool |
Preserve the file owner's GID. Default: false.( --preserve-gid)(-j) |
preserve_modification_time |
Bool |
Preserve file modification time from the source to the destination. Default: false.( --preserve-modification-time) |
preserve_object_lock_legal_hold |
Bool |
Preserve object lock legal hold status from the source to the destination. Default: false.( --preserve-object-lock-legal-hold) |
preserve_object_lock_retention |
Bool |
Preserve object lock retention from the source to the destination. Default: false.( --preserve-object-lock-retention) |
preserve_object_metadata |
Bool |
Preserve object metadata from the source to the destination. Default: false.( --preserve-object-metadata) |
preserve_uid |
Bool |
Preserve the file owner's UID. Default: false.( --preserve-uid)(-u) |
quiet |
Bool |
Disable progress display. Default: true.( --quiet)(-q) |
remote |
Hash |
Required. |
remote.connect_mode |
Enum[String] |
Define how to connect to the remote. Allowed values: ssh, ws.Default: ssh.(special: --ws-connect) |
remote.fingerprint |
String |
Check it against server SSH host key fingerprint. |
remote.host |
String |
Use the specified host name or address of the remote host. Required. ( --host={string}) |
remote.pass |
String |
Authenticate the transfer with the specified password. ( --pass={string})(-w) |
remote.path |
String |
Synchronize the specified directory on the remote host. Required. ( --remote-dir={string})(-r) |
remote.port |
Integer |
Use the specified TCP port for SSH. Used when connect_mode is sshDefault: 22.( --tcp-port={integer})(-P) |
remote.private_key_paths |
Array[String] |
Authenticate with the specified SSH private key file. ( --private-key-path={array})(-i) |
remote.proxy |
Hash |
Specify the address of the Aspera high-speed proxy server. (special: --proxy={object}) |
remote.proxy.host |
String |
Use the specified host name or address of the proxy. |
remote.proxy.pass |
String |
Authenticate to the proxy with the specified password. |
remote.proxy.port |
Integer |
Use the specified port. Default is 9091 for dnat, 9092 for dnats. |
remote.proxy.protocol |
Enum[String] |
The protocol to be used. Allowed values: none, dnat, dnats.Default: none. |
remote.proxy.user |
String |
Authenticate to the proxy with the specified username. |
remote.token |
String |
Token string passed to server's authentication service. |
remote.token_node_user |
String |
Node API user identity associated with the token. Required for node user bearer tokens |
remote.user |
String |
Authenticate the transfer with the specified username. Required. ( --user={string}) |
remote.ws_port |
Integer |
Use the specified port for Websocket. Used when connect_mode is ws.Default: 9093. |
remote_checksum_threads |
Integer |
Maximum number of threads to do checksum on the remote host. Value must be between 1 and 99 Default: 4.( --remote-checksum-threads={integer}) |
remote_db_dir |
String |
Use the specified database directory on the remote host. Default is .private-asp at the root level of the synchronized directory.( --remote-db-dir={string})(-B) |
remote_db_store_dir |
String |
Store/Restore the database to/from the specified directory on the remote host. The value can be an absolute path, an URI or - (use the remote sync dir). ( --remote-db-store-dir={string}) |
remote_force_stat |
Bool |
Forces the remote async to retrieve file information even when no changes are detected by the scanner or monitor. Default: false.( --remote-force-stat) |
remote_fs_threads |
Integer |
Maximum number of threads to do file system operations on the remote host. Value must be between 1 and 99. Default: 1.( --remote-fs-threads={integer}) |
remote_keep_dir |
String |
Move deleted files into the specified directory on the remote host. ( --keep-dir-remote={string}) |
remote_mount_signature |
String |
Verify that the file system is mounted by the existence of this file on the remote host. ( --remote-mount-signature={string}) |
remote_move_cache_timeout_seconds |
Integer |
Delay in seconds before aborting moving a file from remote cache to final destination. -1 for disabled.Default: -1.( --remote-move-cache-timeout={integer}) |
remote_preserve_acls |
Enum[String] |
Preserve access control lists on the remote host. If not specified, the default behavior is to use the same storage mode as specified by preserve_acls.Allowed values: native, metafile, none.( --remote-preserve-acls={enum}) |
remote_preserve_xattrs |
Enum[String] |
Preserve extended attributes on the remote host. If not specified, the default behavior is to use the same storage mode as specified by preserve_xattrs.Allowed values: native, metafile, none.( --remote-preserve-xattrs={enum}) |
remote_scan_interval_milliseconds |
Integer |
Enable periodic scans on the remote host. -1 for disabled. On the command line, the value is a duration, e.g. 1h 30m, 2s500ms, 5:30, 1d 02:00:00.250. A plain number is in seconds. Stored in milliseconds.Default: -1.(special: --remote-scan-interval={integer}) |
remote_scan_threads |
Integer |
Number of directory scanning threads on the remote host. Value must be between 1 and 99. Default: 1.( --remote-scan-threads={integer}) |
remote_stat_cache_size |
Integer |
Set stat cache size on the remote host. 0 for disabled. Default: 0. |
remove_after_transfer |
Bool |
Remove source files after they are successfully synchronized. Default: false.( --remove-after-transfer) |
reset |
Bool |
Clear the snapshot database and rescan the synchronized directories and files to create a fresh snapshot Default: false.( --reset)(-x) |
resume |
Hash |
Partial transfers may exist if communication disruptions caused the underlying ascp processes to terminate early. Note that transfer resumption can only happen if the reset option is disabled. If an async session starts with reset enabled and resume enabled, transfers interrupted during that session will be resumable, but only if async is then restarted with 'reset' disabled. |
resume.enabled |
Bool |
Enable the possibility of resuming individual file transfers between async sessions. Default: false. |
resume.max_age |
Integer |
Sets the age limit in days for temporary files that will be preserved on cleanup (usually at async's start and stop) for potential transfer resume. Temp files older than the given value will be removed regardless of whether they might be resumable. Default: 5.( --resume-age-days={integer}) |
resume.min_size |
Integer |
This field specifies the minimum size of files that will be allowed to resume. Default: 1048576.( --support-resume={integer}) |
resume_scan |
Bool |
Resume the scan from where the previous execution left off. Default: false.( --resume-scan) |
scan_dir_rename |
Bool |
Enable the detection of renamed directories and files compared to the previous scan, based on matching inodes Default: false.( --scan-dir-rename) |
scan_file_rename |
Bool |
Enable the detection of renamed files compared to the previous scan, based on matching inodes. Default: false.( --scan-file-rename) |
scan_intensity |
Enum[String] |
Scan at the set intensity. vlow minimizes system activity. vhigh maximizes system activity by continuously scanning files without rest.Allowed values: vlow, low, medium, high, vhigh.Default: medium.( --scan-intensity={enum})(-H) |
sharing_retry_max |
Integer |
Retry synchronizations up to the specified maximum number after a sharing violation. Default: 3.( --sharing-retry-max={integer}) |
store_metadata_records |
Bool |
Store the acls or xattrs in the snapshot database. Default: false.( --store-metadata-records) |
symbolic_links |
Enum[String] |
Handle symbolic links with the specified method. Default is skip on windows, copy otherwise.Allowed values: copy, skip, follow.( --symbolic-links={enum})(-n) |
tags |
Hash |
User-defined metadata tags. (special: --tags64={object}) |
transfer_threads |
Array[Hash] |
Use the specified number of dedicated transfer threads to process files smaller or equal to the specified size (special: --transfer-threads={array}) |
transfer_threads[].size |
Integer |
Upper limit. -1 for infinity.Default: -1. |
transfer_threads[].threads |
Integer |
The number of threads. Required. |
transport |
Hash |
|
transport.cipher |
Enum[String] |
Specify encryption algorithm for file data. Allowed values: none, aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm.Default: aes128.( --cipher={enum})(-c) |
transport.compression |
Enum[String] |
Compress a file before transfer using the specified MODE. Allowed values: none, zlib.Default: none.( --compression={enum}) |
transport.datagram_size |
Integer |
Specify the datagram size (MTU) for FASP. By default it uses the detected path MTU. ( --datagram-size={integer})(-Z) |
transport.min_rate |
Integer |
Attempt to transfer no slower than the specified rate (in bps). Default: 0.( --min-rate={integer})(-m) |
transport.rate_policy |
Enum[String] |
Defines how ascp will manage the bandwidth.Allowed values: fair, fixed, high, low.Default: fair.( --rate-policy={enum})(-a) |
transport.raw_options |
Array[String] |
Pass arbitrary arguments to ascp.(special: --raw-options={array}) |
transport.read_block_size |
Integer |
Use the specified block size (in bytes) for reading. Default is determined by aspera.conf.( --read-block-size={integer})(-g) |
transport.rexmsg_size |
Integer |
Use the specified size (in bytes) for a retransmission request. Default is determined by aspera.conf.( --rexmsg-size={integer})(-X) |
transport.target_rate |
Integer |
Transfer no faster than the specified rate (in bps). A string with suffix k, m or g is converted to bps by ascli (e.g. 100m).Default: 10000000.( --target-rate={integer})(-l) |
transport.udp_port |
Integer |
Use the specified UDP port for FASP data transfer. Default: 33001.( --udp-port={integer})(-O) |
transport.write_block_size |
Integer |
Use the specified block size (in bytes) for writing. Default is determined by aspera.conf.( --write-block-size={integer})(-G) |
watchd |
Hash |
When connection is configured, asperawatchd is used to detect the changes on the source directory.(special: --watchd={object}) |
watchd.datastore |
Enum[String] |
Specify the type of datastore, none for disabled.Allowed values: none, redis, scalekv.Default: none. |
watchd.domain |
String |
Specify the domain. Default is the current username. |
watchd.host |
String |
Use the specified host name or address to connect to the datastore. |
watchd.port |
Integer |
Use the specified port. Default: 31415. |
write_gid |
String |
Try to write files as the specified group. ( --write-gid={string}) |
write_uid |
String |
Try to write files as the specified user. ( --write-uid={string}) |
sync_info: args format🔗
This is the legacy syntax.
ascli defines a JSON equivalent to regular async options.
It is based on a JSON representation of async command line options.
Technically, it allows definition of multiple sync sessions in a single command, but ascli only accepts a single session for consistency with the previous syntax.
This format is selected if the sync_info Hash has either key sessions or instance.
The following parameters are automatically filled from mandatory arguments, and are not allowed:
directionlocal_dirremote_dir
Parameter name is set to a default value if not provided in sync_info.
Sync management and monitoring: admin🔗
The admin command provides subcommands to inspect the state of an Async sync session.
Most subcommands read the local snap database (snap.db) directly - no server connection is required.
The exception is status, which calls the asyncadmin utility available only on server products.
These commands can also be run from the config plugin:
ascli config sync admin
Prerequisites: Most admin subcommands require the sqlite3 gem:
gem install sqlite3
Usage: All admin subcommands share the same argument structure:
ascli ... sync admin <COMMAND> <FOLDER> [<SYNC_INFO>]
<COMMAND>- one of the subcommands listed below.<FOLDER>- path to the local database folder (a folder containing a.private-aspsubfolder). By default this is the local synchronized folder; if a separate database folder was configured, specify that path instead.[<SYNC_INFO>]- optionalHashto identify the session. If the folder contains only one session it is selected automatically; otherwise provide@: name=<SESSION_NAME>.
The only exception is find, which takes a plain directory path and lists all <SESSION_NAME> found inside it.
Subcommands:
| Command | Uses | Description |
|---|---|---|
find |
filesystem | Search a directory for existing sync sessions and list their names and paths |
status |
asyncadmin |
Retrieve the status of a running sync session (requires a server-side product) |
meta |
sqlite3 |
Display session metadata from sync_snapmeta_table |
counters |
sqlite3 |
Display synchronization counters from sync_snap_counters_table |
file_info |
sqlite3 |
List the synchronization state of each file from sync_snapdb_table (state, record ID, path, message) |
overview |
sqlite3 |
List all tables and their columns in the snap database |
query |
sqlite3 |
Execute an arbitrary SQL statement against the snap database and return the result rows |
The query subcommand requires --sql with the full SQL statement to execute.
The sqlite3-based subcommands (meta, counters, file_info) accept --sql optionally to append a SQL fragment (for example, a WHERE or ORDER BY clause):
ascli ... sync admin query <FOLDER> --sql="SELECT * FROM sync_snapdb_table WHERE state=20"
ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20"
ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20 ORDER BY f_meta_path"
Snap database schema: The snap database (snap.db) contains the following tables:
sync_snapmeta_table - one row per session, written at start and updated at stop:
| Field | Type | Description |
|---|---|---|
ascp_sids |
String |
Last known FASP (ascp) session IDs. |
byte_peerxfer |
Integer |
Number of bytes transferred by the peer. |
byte_synced |
Integer |
Number of bytes successfully synchronized. |
byte_xferres |
Integer |
Number of bytes for which transfer was resumed. |
checksum |
Enum[String] |
Checksum algorithm. Allowed values: sha1, md5, sha1_sparse, md5_sparse, none. |
cip_true_val |
Integer |
Candidate-in-progress counter true value (internal). |
condition |
Enum[String] |
Last known session condition. Allowed values: running, stopped. |
dbversion |
String |
Database schema version. |
direction |
Enum[String] |
Sync direction. Allowed values: push, pull, bidi. |
exclude_dirs_older_than |
Integer |
Exclude directories with modification time older than this epoch value (0 = disabled). |
file_peerxfer |
Integer |
Number of files transferred by the peer. |
file_synced |
Integer |
Number of files successfully synchronized. |
file_xferres |
Integer |
Number of files for which transfer was resumed. |
filters |
String |
Active include/exclude filter patterns. |
inode_format |
String |
Inode format used by the session. |
ldir |
String |
Local directory path. |
max_recid |
Integer |
Highest record ID allocated in the snapshot table. |
pairname |
String |
Sync session name. |
path_peerxfer |
Integer |
Number of paths transferred by the peer. |
path_synced |
Integer |
Number of paths successfully synchronized. |
path_xferres |
Integer |
Number of paths for which transfer was resumed. |
rdir |
String |
Remote directory path. |
rhost |
String |
Remote host. |
scn_true_val |
Integer |
Scan counter true value (internal). |
start_time |
String |
Session start timestamp. |
stop_time |
String |
Session stop timestamp (written when session ends). |
sync_point |
Integer |
Synchronization point sequence number. |
sync_uuid |
String |
UUID identifying this sync pair. |
sync_snap_counters_table - one row, updated live during a session:
| Field | Type | Description |
|---|---|---|
conflict |
Integer |
Number of paths in the Conflict state. |
error |
Integer |
Number of paths in the Error state. |
files_conflict |
Integer |
Number of files in the Conflict state. |
files_error |
Integer |
Number of files in the Error state. |
files_pending |
Integer |
Number of files still pending synchronization. |
files_potential_conflict |
Integer |
Number of files in the Potential conflict state. |
files_syncd |
Integer |
Number of files in the Syncd state. |
pending |
Integer |
Number of paths still pending synchronization. |
potential_conflict |
Integer |
Number of paths in the Potential conflict state. |
syncd |
Integer |
Number of paths in the Syncd state. |
unused |
Integer |
Reserved (unused counter slot). |
sync_snapdb_table - one row per tracked file or directory:
| Field | Type | Description |
|---|---|---|
c_cont_csum |
String |
Current content — checksum. |
c_cont_mtime |
Integer |
Current content — modification time (epoch). |
c_cont_size |
Integer |
Current content — file size in bytes. |
c_cont_timestamp |
Integer |
Current content — internal timestamp. |
c_meta_atime |
Integer |
Current metadata — access time (epoch). |
c_meta_attrs |
String |
Current metadata — file attributes. |
c_meta_ctime |
Integer |
Current metadata — change time (epoch). |
c_meta_gid |
Integer |
Current metadata — group ID (Unix). |
c_meta_mtime |
Integer |
Current metadata — modification time (epoch). |
c_meta_object_metadata |
String |
Current metadata — object metadata (JSON). |
c_meta_ol_legal_hold |
Integer |
Current metadata — object lock legal hold flag. |
c_meta_ol_mode |
Integer |
Current metadata — object lock mode. |
c_meta_ol_retain_until |
Integer |
Current metadata — object lock retain-until date (epoch). |
c_meta_parentrid |
Integer |
Current metadata — parent record ID. |
c_meta_path |
String |
Current metadata — file path as known to the snapshot. |
c_meta_type |
Integer |
Current metadata — file type (file, directory, …). |
c_meta_uid |
Integer |
Current metadata — user ID (Unix). |
c_meta_xattrs |
Integer |
Current metadata — extended attributes (not used). |
c_meta_xattrs_cksum |
String |
Current metadata — checksum of extended attributes. |
candidate_in_prog |
Integer |
Set to 1 when this record is currently in the candidate transfer list. |
commit_version |
Integer |
Local commit version for this entry. |
f_cont_csum |
String |
Filesystem content — checksum. |
f_cont_mtime |
Integer |
Filesystem content — modification time (epoch). |
f_cont_size |
Integer |
Filesystem content — file size in bytes. |
f_cont_timestamp |
Integer |
Filesystem content — internal timestamp. |
f_meta_atime |
Integer |
Filesystem metadata — access time (epoch). |
f_meta_attrs |
String |
Filesystem metadata — file attributes. |
f_meta_ctime |
Integer |
Filesystem metadata — change time (epoch). |
f_meta_gid |
Integer |
Filesystem metadata — group ID (Unix). |
f_meta_mtime |
Integer |
Filesystem metadata — modification time (epoch). |
f_meta_object_metadata |
String |
Filesystem metadata — object metadata (JSON). |
f_meta_ol_legal_hold |
Integer |
Filesystem metadata — object lock legal hold flag. |
f_meta_ol_mode |
Integer |
Filesystem metadata — object lock mode. |
f_meta_ol_retain_until |
Integer |
Filesystem metadata — object lock retain-until date (epoch). |
f_meta_parentrid |
Integer |
Filesystem metadata — parent record ID. |
f_meta_path |
String |
Filesystem metadata — file path as seen on disk (displayed by file_info). |
f_meta_type |
Integer |
Filesystem metadata — file type. |
f_meta_uid |
Integer |
Filesystem metadata — user ID (Unix). |
f_meta_xattrs |
Integer |
Filesystem metadata — extended attributes (not used). |
f_meta_xattrs_cksum |
String |
Filesystem metadata — checksum of extended attributes. |
message |
String |
Error or status message associated with this entry. |
meta_lastreq_no |
Integer |
Sequence number of the last metadata request sent. |
meta_lastresp_no |
Integer |
Sequence number of the last metadata response received. |
nodeid |
Integer |
Local filesystem node ID. |
p_cont_csum |
String |
Pending content — checksum. |
p_cont_mtime |
Integer |
Pending content — modification time (epoch). |
p_cont_size |
Integer |
Pending content — file size in bytes. |
p_cont_timestamp |
Integer |
Pending content — internal timestamp. |
pcr_reqno |
Integer |
Peer content request sequence number. |
peer_commit_version |
Integer |
Peer commit version for this entry. |
peer_nodeid |
Integer |
Peer filesystem node ID. |
peer_rid |
Integer |
Peer record ID for this entry. |
pp_cont_csum |
String |
Peer pending content — checksum. |
pp_cont_mtime |
Integer |
Peer pending content — modification time (epoch). |
pp_cont_size |
Integer |
Peer pending content — file size in bytes. |
pp_cont_timestamp |
Integer |
Peer pending content — internal timestamp. |
record_id |
Integer |
Primary key (auto-increment). |
recursive_mtime |
Integer |
Most recent modification time across all descendants (directories only). |
scn_completed |
Integer |
Set to 1 when the scan of this directory entry has completed. |
scn_not_visited |
Integer |
Set to 1 when the scanner has not yet visited this entry in the current scan pass. |
scn_required |
Integer |
Set to 1 when a rescan of this entry is required. |
state |
Integer |
Sync state of the entry. | Value | Name | |-------|-------------------| | 0 | Nil | | 1–18 | Pending (variant) | | 19 | Syncd | | 20 | Error | | 21 | Conflict | | 22 | PotentialConflict | | 23–24 | Pending (variant) | |
tobe_deleted |
Integer |
Set to 1 when the record is flagged for deletion (garbage collection pending). |
version |
Integer |
Schema version field (not used). |
Hot folder🔗
ascli does not have a standalone hotfolder command.
Instead, you create a Hot Folder by combining the upload or download commands with specific transfer-spec parameters and an OS scheduler (like cron or Task Scheduler).
Requirements🔗
ascli may be used as a simple hot folder engine.
A hot folder is defined as a tool that:
- Locally (or remotely) detects new files in a top folder
- Sends detected files to a remote (respectively, local) repository
- Only sends new files, and does not re-send already sent files
- Optionally: sends only files that are not still growing
- Optionally: after transfer of files, deletes or moves to an archive
In addition: the detection should be made continuously or on specific time/date.
Setting up a hot folder🔗
The general idea is to rely on:
- Existing
ascpfeatures for detection and transfer ascliconfiguration capabilities and server-side knowledge- The OS scheduler for reliability and continuous operation
ascp features🔗
Useful ascp features are available as arguments (see the ascp manual):
- Sending only new files
- Option
-k 1,2,3(resume_policy) - Remove or move files after transfer:
--remove-after-transfer(remove_after_transfer)--move-after-transfer(move_after_transfer)--remove-empty-directories(remove_empty_directories)- Send only files not modified since the last X seconds:
--exclude-newer-than(exclude_newer_than)--exclude-older-than(exclude_older_than)- Top level folder should not be created on destination
--src-base(src_base)
Note
ascli takes transfer parameters exclusively as a transfer-spec, with ts option.
Note
Usual native ascp arguments are available as standard transfer-spec parameters, but not special or advanced options.
Tip
Only for the direct transfer agent (not others, like connect or node), native ascp arguments can be provided with parameter ascp_args of option transfer.
Server side and configuration🔗
Virtually any transfer on a repository on a regular basis might emulate a hot folder.
Note
File detection is not based on events (inotify, and so on), but on a simple folder scan on source side.
Tip
Options may be saved in an Option Preset and used with -P.
Scheduling🔗
Once ascli command line arguments are defined, run the command using the OS native scheduler, for example, every minute or every 5 minutes.
See Scheduler (and option lock_port).
Example: Upload hot folder🔗
ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than":-8,"src_base":"source_hot"}'
The local folder (here, relative path: source_hot) is sent (upload) to an Aspera server.
Source files are deleted after transfer.
Growing files will be sent only once they do not grow anymore (based on an 8-second cool-off period).
If a transfer takes more than the execution period, then the subsequent execution is skipped (lock_port) preventing multiple concurrent runs.
Example: Unidirectional synchronization (upload) to server🔗
ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'
This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node.
Example: Unidirectional synchronization (download) from Aspera on Cloud Files🔗
ascli aoc files download . --to-folder=. --lock-port=12345 --progress-bar=no --out.level=data --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'
Note
Option delete_before_transfer will delete files locally, if they are not present on remote side.
Note
Options progress_bar and --out.level limit output for headless operation (for example, a cron job).
Health check and Nagios🔗
Most plugins provide a health command that checks the health status of the application.
Example:
ascli console health
╭────────┬─────────────┬────────────╮
│ status │ component │ message │
╞════════╪═════════════╪════════════╡
│ ok │ console api │ accessible │
╰────────┴─────────────┴────────────╯
Typically, the health check uses the REST API of the application, with the following exception: the server plugin checks health by issuing a transfer to the server (server health transfer).
ascli can be called by Nagios to check the health status of an Aspera server.
The output can be made compatible with Nagios with option --format=nagios:
ascli server health transfer --to-folder=/Upload --format=nagios --progress-bar=no
OK - [transfer:ok]
SMTP for email notifications🔗
ascli can send emails.
To do so, set up the SMTP configuration with option smtp.
The smtp option is a Hash (Extended Value) with the following fields:
| Field | Type | Description |
|---|---|---|
domain |
String |
Email domain of the user. Defaults to the domain part of server.Example: gmail.com. |
from_email |
String |
Sender address (used in From: header and as reply-to). Defaults to username when defined.Example: johnny@example.com. |
from_name |
String |
Display name of the sender. Defaults to the local-part of from_email.Example: John Wayne. |
password |
String |
Password for the above username.Example: <PASSWORD>. |
port |
Integer |
Port for the SMTP service. Defaults: 587 (TLS), 465 (SSL), 25 (plain).Example: 587. |
server |
String |
SMTP server address. Required. Example: smtp.gmail.com. |
ssl |
Bool |
Enable TLS/SSL (port 465).Default: false. |
tls |
Bool |
Enable STARTTLS (port 587).Default: true. |
username |
String |
User to authenticate on the SMTP server. Leave empty for open (unauthenticated) relay. Example: john@example.com. |
Example of configuration🔗
ascli config preset set smtp_google server smtp.google.com
ascli config preset set smtp_google username john@gmail.com
ascli config preset set smtp_google password <PASSWORD>
or
ascli config preset initialize smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"<PASSWORD>"}'
or
ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=<PASSWORD>
Set this configuration as global default, for instance:
ascli config preset set cli_default smtp @val:@preset:smtp_google
ascli config preset set default config cli_default
Email templates🔗
Sent emails are built using a template that uses the ERB syntax.
The template is the full SMTP message, including headers.
The following variables are defined by default:
from_namefrom_emailto
Other variables are defined depending on context.
Test🔗
Check settings with smtp_settings command.
Send test email with email_test.
ascli config smtp_settings --smtp=@preset:smtp_google
ascli config email_test --smtp=@preset:smtp_google --notify-to=sample.dest@example.com
Notifications for transfer status🔗
An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).
To activate, use option notify_to.
A default e-mail template is used, but it can be overridden with option notify_template.
The environment provided contains the following additional variables:
subject: a default subject including transfer statusstatus: global status of transferts: the transfer-spec used for the transferfrom_email: email of sender (fromsmtpconfiguration)from_name: name of sender (fromsmtpconfiguration)to: recipient of the email (fromnotify_to)
Example of template:
From: <%=from_name%> <<%=from_email%>>
To: <<%=to%>>
Subject: <%=subject%>
Transfer is: <%=status%>
Tool: asession🔗
This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: asession.
It aims at simplifying the startup of a FASP session from a programmatic standpoint as formatting a transfer-spec is:
- Common to Aspera Node API (HTTP POST /ops/transfer)
- Common to Aspera Connect API (browser JavaScript
startTransfer) - Generated by using any third-party language-specific JSON library
Ideally, IBM will integrate this directly into ascp, making this tool redundant.
Integration with any language is possible, provided that the language can spawn a subprocess, write to its STDIN, read from STDOUT, and generate and parse JSON.
asession expects a single argument: a session specification that contains parameters and a transfer-spec.
If no argument is provided, it assumes a value of: @json:@stdin:, that is, JSON on stdin.
Note
If JSON is the format, specify @json: to tell asession to decode the Hash using JSON syntax.
During execution, it generates all low level events, one per line, in JSON format on stdout.
Top level parameters supported by asession:
| Parameter | Description |
|---|---|
spec |
The transfer-spec |
agent |
Same parameters as transfer option for agent direct |
loglevel |
Log level of asession |
file_list_folder |
The folder used to store (for garbage collection) generated file lists. Default: [system tmp folder]/[username]_asession_filelists |
Comparison of interfaces🔗
| Feature/Tool | Transfer Daemon | FASPManager | ascp |
asession |
|---|---|---|---|---|
| status | Supported | Deprecated | Supported | Deprecated |
| language integration | Many | C/C++ C#/.net Go Python java |
Any | Any |
required additional components to ascp |
Daemon | Library (+headers) |
- | Ruby Aspera gem |
| startup | Daemon | API | Command line arguments | JSON on stdin (standard APIs: JSON.generate Process.spawn) |
| events | Poll | Callback | Possibility to open management port and proprietary text syntax |
JSON on stdout |
| platforms | Like ascp and transferd |
Like ascp and lib (if compiled) |
Any with ascp |
Any with Ruby and ascp |
Simple session🔗
Create a file session.json with:
{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}
Then start the session:
asession < session.json
Asynchronous commands and Persistent session🔗
asession also supports asynchronous commands (on the management port).
Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be snake style, for example: LongParameter → long_parameter
This is particularly useful for a persistent session (with the transfer-spec parameter: "keepalive":true)
asession
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"<PASSWORD>","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
{"type":"DONE"}
(events from FASP are not shown in above example. They would appear after each command)
Help🔗
asession -h
USAGE
asession
asession -h|--help
asession [<session spec extended value>]
If no argument is provided, default will be used: @json:@stdin
-h, --help display this message
<session spec extended value> a dictionary (Hash)
The value can be either:
the JSON description itself, e.g. @json:'{"xx":"yy",...}'
@json:@stdin, if the JSON is provided from stdin
@json:@file:<path>, if the JSON is provided from a file
The following keys are recognized in session spec:
spec : mandatory, contains the transfer spec
loglevel : modify log level (to stderr)
agent : modify transfer agent parameters, e.g. ascp_args
file_list_folder : location of temporary files
sdk : location of SDK (ascp)
Asynchronous commands can be provided on STDIN, examples:
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
{"type":"START","source":"xx","destination":"yy"}
{"type":"DONE"}
EXAMPLES
asession @json:'{"spec":{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}}'
echo '{"spec":{"remote_host":...}}'|asession @json:@stdin:
Ruby Module: Aspera🔗
Main components:
Asperageneric classes for REST and OAuthAspera::Agent::Direct: Starting and monitoring transfers usingascp.Aspera::Cli:ascli.
Working examples can be found in repo: https://github.com/laurent-martin/aspera-api-examples in Ruby examples.