Skip to content

Operational Utilities🔗

This section covers the specialized modules and utilities used to integrate ascli into your broader operational infrastructure. While the core plugins handle data movement, these tools provide the "integration layer" for enterprise environments: Aspera Sync and Hot Folder enable automated, folder-based synchronization; Nagios and SMTP modules provide health monitoring and automated email alerting for transfer status; and the asession tool and the Ruby module Aspera allow integration of transfers into other programs. Together, these features transform the CLI from a manual tool into a fully integrated component of an automated, monitored data workflow.

IBM Aspera Sync🔗

An interface for the async utility is provided in the following plugins:

  • server sync (SSH Auth)
  • node sync (use gen3 token)
  • aoc files sync (uses node plugin with bearer token)
  • shares files sync (uses node plugin with gen3 token)

The sync command, available in above plugins, performs the following actions:

  • Start a local Sync session by executing the async command with the appropriate parameters.
  • Get local Sync session information by accessing the Async snap database directly.
  • Get local Sync session information using the asyncadmin command, if available.

One advantage of using ascli over the async command line is the possibility to use a configuration file, using standard options of ascli. Moreover, ascli supports sync with applications requiring token-based authorization.

Some sync parameters are filled by the related plugin using transfer spec parameters (for example, including token).

Note

All sync commands require an async enabled license and availability of the async executable (and asyncadmin). The Aspera Transfer Daemon 1.3+ includes this.

Quick test🔗

Below is a simple end-to-end procedure to verify synchronization using the demo server.

  1. Initialize the demo server configuration:

    ascli config initdemo
    
  2. Create a local folder with a sample file to sync:

    mkdir foobar
    echo hello > foobar/file1
    
  3. Create the destination folder on the remote server:

    ascli server -Pdemoserver mkdir /Upload/mydest1
    
  4. Run the sync operation:

    ascli server -Pdemoserver sync push foobar --to-folder=/Upload/mydest1
    

The following sections provide additional details on available options and configuration.

Starting a sync session🔗

To start a sync session, use one of the three sync directions, followed by a folder path. For push and bidi, the path is a local folder. For pull, the path is a remote folder.

The corresponding path on the opposite side is provided using the to_folder option.

General syntax:

ascli ... sync <DIRECTION> <PATH> [--to-folder=<PATH>] [<SYNC_INFO>]

(parameter)

(parameter)
to_folder
(option)
push Local Remote
bidi Local Remote
pull Remote Local

An optional positional Hash argument (<SYNC_INFO>) may be provided. It can be expressed in one of two formats:

  • conf format (recommended)
  • args format (legacy)

A single sync session must use one format exclusively.

  • If <SYNC_INFO> is not provided, a default configuration is automatically generated in the conf format (details in the next section).

  • If argument <SYNC_INFO> is provided, the format is inferred:

  • If the Hash contains either key sessions or instance, the args format is used.

  • Otherwise, the conf format is used.

sync_info: conf format🔗

This is the preferred syntax. It is the same payload as specified on the async option --conf or in Node API /asyncs.

Documentation on Async Node API can be found on IBM Developer Portal.

The following parameters are automatically filled from mandatory arguments, and are not allowed:

  • direction
  • local.path
  • remote.path

Parameter name is set to a default value if not provided in sync_info. Parameter quiet is set to false if not provided in sync_info and a terminal is detected.

The documentation is available in the terminal with:

ascli config sync spec

Note

ascli accepts the following fields within the sync_info Hash. The options listed in the Description column correspond to the equivalent parameters used by the low-level async command.

Field Type Description
ascp_dir String Directory containing ascp executable to use.
assume_no_mods Bool Assume that the directory structure has not been modified.
Default: false.
(--assume-no-mods)
checksum Enum[String] Use the specified checksum type. Default is none on cloud storage.
Allowed values: sha1, md5, sha1_sparse, md5_sparse, none.
Default: sha1_sparse.
(--checksum={enum})(-k)
clean_excluded Bool Removes any existing entries in the snapshot database for excluded paths
Default: false.
(--clean-excluded)
cookie String User-defined identification string.
(--cookie={string})
cooloff_max_seconds Integer Wait up to the specified time for a file to stop changing before skipping synchronization of the file. 0 for disabled
Default: 0.
(--cooloff-max={integer})
cooloff_seconds Integer Delay the start of the transfer to confirm that the content is not changing. Value must be between 0 and 60
Default: 3.
(--cooloff={integer})
create_dir Bool Create the source directory, target directory, or both, if they do not exist.
Default: false.
(--create-dir)
db_cache_size Integer Specify DB cache size.
Default: 16000.
db_journal_off Bool Turn off DB journal.
Default: false.
db_sync_on Bool Enable synchronous write in DB.
Default: false.
dedup Enum[String] Take the specified action when async detects duplicate files on the source.
Allowed values: copy, inode, hardlink, none.
Default: none.
(--dedup={enum})
delete_before Bool Schedule deletes before transfers.
Default: false.
(--delete-before)
delete_delay Bool Delay actual deletes until the end of the synchronization.
Default: false.
(--delete-delay)
direction Enum[String] The direction of replication relative to the local.
Allowed values: bidi, pull, push.
Default: push.
(--direction={enum})(-K)
exclude_dirs_older_than Hash Don't scan directories with a recursive modified time older than absolute or async start time - relative_seconds
exclude_dirs_older_than.absolute String UTC timestamp. Empty value for disabled.
exclude_dirs_older_than.relative_seconds Integer Relative to async start time. -1 for disabled.
Default: -1.
filters Array[Hash] The filters allow to further specify which files have to be excluded and included from the transfer list. Each filter is defined by a rule and a value. Order of filters matters
filters[].rule Enum[String] The rule for the filter.
Required.
Allowed values: include, exclude, include_from, exclude_from.
filters[].value String On include or exclude, the filter's pattern. On include_from or exclude_from, the path containing filter specifications
Required.
ignore_delete Bool Do not copy removals to the peer.
Default: false.
(--ignore-delete)
ignore_mode Bool Source files that have had their mode changed after the initial transfer will not update the destination file mode.
Default: false.
(--ignore-mode)
ignore_remote_host_sync_name Bool Do not check that the remote host being used for the current transfer matches the host used when the local database was created
Default: false.
local Hash Required.
local.pass String Authenticate the local async with the specified password.
local.path String The directory to be synchronized on the local host.
Required.
(--local-dir={string})(-d)
local_apply_docroot Bool Prepend the docroot to the directory on the local host.
Default: false.
(--apply-local-docroot)
local_checksum_threads Integer Maximum number of threads to do checksum on the local host. Value must be between 1 and 99.
Default: 4.
(--local-checksum-threads={integer})
local_db_dir String Use the specified database directory on the local host. Default is .private-asp at the root level of the synchronized directory.
(--local-db-dir={string})(-b)
local_db_store_dir String Store/Restore the database to/from the specified directory on the local host. The value can be an absolute path, an URI or - (use the local sync dir)
(--local-db-store-dir={string})
local_force_stat Bool Forces the local async to retrieve file information even when no changes are detected by the scanner or monitor.
Default: false.
(--local-force-stat)
local_fs_threads Integer Maximum number of threads to do file system operations on the local host. Value must be between 1 and 99.
Default: 1.
(--local-fs-threads={integer})
local_keep_dir String Move deleted files into the specified directory on the local host.
(--keep-dir-local={string})
local_mount_signature String Verify that the file system is mounted by the existence of this file on the local host.
(--local-mount-signature={string})
local_move_cache_timeout_seconds Integer Delay in seconds before aborting moving a file from local cache to final destination. -1 for disabled.
Default: -1.
(--local-move-cache-timeout={integer})
local_preserve_acls Enum[String] Preserve access control lists on the local host.
Allowed values: native, metafile, none.
Default: none.
(--preserve-acls={enum})
local_preserve_xattrs Enum[String] Preserve extended attributes on the local.
Allowed values: native, metafile, none.
Default: none.
(--preserve-xattrs={enum})
local_scan_interval_milliseconds Integer Enable periodic scans on the local host during a continuous sync. -1 for disabled. On the command line, the value is a duration, e.g. 1h 30m, 2s500ms, 5:30, 1d 02:00:00.250. A plain number is in seconds. Stored in milliseconds.
Default: -1.
(--scan-interval={integer})
local_scan_threads Integer Number of directory scanning threads on the local host. Value must be between 1 and 99
Default: 1.
(--scan-threads={integer})
local_stat_cache_size Integer Set stat cache size on the local host. 0 for disabled.
Default: 0.
log Hash
log.level Enum[String] Use the specified log level.
Allowed values: log, dbg1, dbg2.
Default: log.
(special:-D)
log.local_dir String Use the specified logging directory on the local host.
(--alt-logdir={string})(-L)
log.remote_dir String Use the specified logging directory on the remote host.
(--remote-logdir={string})(-R)
manifest_path String A directory path where ascp will create manifest TEXT files (passed to ascp as --file-manifest-path)
mirror Bool Force the pulling side to be exactly like the pushing side, removing files on the destination that don't exist on the source and resending source files that don't have an exact match on the destination. Cannot be used in bi-directional mode.
Default: false.
(--mirror)
mode Enum[String] Specify whether async runs continuously or not. In one_time mode, async stops after the first full synchronization. continuous supported only if the source is Windows or Linux.
Allowed values: one_time, continuous.
Default: one_time.
(special:--continuous)(-C)
monitor_buffer_size Integer Bytes to allocate for the change monitor buffer. Applies to any Windows machine on either side. -1 to use the computed value.
Default: -1.
name String Name of the synchronization pair.
Required.
(--name={string})(-N)
no_log String Suppress log messages for ITEM. The only currently supported ITEM is 'stats', which suppresses both STATS and PROG log messages.
(--no-log={string})
no_preserve_root_attrs Bool Disable the preservation of attributes on the Sync root.
Default: false.
(--no-preserve-root-attrs)
no_scan Bool Skip initial scanning.
Default: false.
(--no-scan)
notifications_sharing_retry_max Integer Retry processing filesystem notifications up to the specified maximum number after a sharing violation.
Default: 3.
overwrite Enum[String] Overwrite files according to the specified policy. Default is determined by the direction: conflict for bidi, otherwise always.
Allowed values: always, older, conflict.
(--overwrite={enum})(-o)
pending_max Integer Allow the maximum number of files that are pending transfer to be no more than the specified number.
Default: 2000.
(--pending-max={integer})
preserve_access_time Bool Preserve file access time from the source to the destination.
Default: false.
(--preserve-access-time)
preserve_creation_time Bool Preserve file creation time from the source to the destination.
Default: false.
(--preserve-creation-time)
preserve_gid Bool Preserve the file owner's GID.
Default: false.
(--preserve-gid)(-j)
preserve_modification_time Bool Preserve file modification time from the source to the destination.
Default: false.
(--preserve-modification-time)
preserve_object_lock_legal_hold Bool Preserve object lock legal hold status from the source to the destination.
Default: false.
(--preserve-object-lock-legal-hold)
preserve_object_lock_retention Bool Preserve object lock retention from the source to the destination.
Default: false.
(--preserve-object-lock-retention)
preserve_object_metadata Bool Preserve object metadata from the source to the destination.
Default: false.
(--preserve-object-metadata)
preserve_uid Bool Preserve the file owner's UID.
Default: false.
(--preserve-uid)(-u)
quiet Bool Disable progress display.
Default: true.
(--quiet)(-q)
remote Hash Required.
remote.connect_mode Enum[String] Define how to connect to the remote.
Allowed values: ssh, ws.
Default: ssh.
(special:--ws-connect)
remote.fingerprint String Check it against server SSH host key fingerprint.
remote.host String Use the specified host name or address of the remote host.
Required.
(--host={string})
remote.pass String Authenticate the transfer with the specified password.
(--pass={string})(-w)
remote.path String Synchronize the specified directory on the remote host.
Required.
(--remote-dir={string})(-r)
remote.port Integer Use the specified TCP port for SSH. Used when connect_mode is ssh
Default: 22.
(--tcp-port={integer})(-P)
remote.private_key_paths Array[String] Authenticate with the specified SSH private key file.
(--private-key-path={array})(-i)
remote.proxy Hash Specify the address of the Aspera high-speed proxy server.
(special:--proxy={object})
remote.proxy.host String Use the specified host name or address of the proxy.
remote.proxy.pass String Authenticate to the proxy with the specified password.
remote.proxy.port Integer Use the specified port. Default is 9091 for dnat, 9092 for dnats.
remote.proxy.protocol Enum[String] The protocol to be used.
Allowed values: none, dnat, dnats.
Default: none.
remote.proxy.user String Authenticate to the proxy with the specified username.
remote.token String Token string passed to server's authentication service.
remote.token_node_user String Node API user identity associated with the token. Required for node user bearer tokens
remote.user String Authenticate the transfer with the specified username.
Required.
(--user={string})
remote.ws_port Integer Use the specified port for Websocket. Used when connect_mode is ws.
Default: 9093.
remote_checksum_threads Integer Maximum number of threads to do checksum on the remote host. Value must be between 1 and 99
Default: 4.
(--remote-checksum-threads={integer})
remote_db_dir String Use the specified database directory on the remote host. Default is .private-asp at the root level of the synchronized directory.
(--remote-db-dir={string})(-B)
remote_db_store_dir String Store/Restore the database to/from the specified directory on the remote host. The value can be an absolute path, an URI or - (use the remote sync dir).
(--remote-db-store-dir={string})
remote_force_stat Bool Forces the remote async to retrieve file information even when no changes are detected by the scanner or monitor.
Default: false.
(--remote-force-stat)
remote_fs_threads Integer Maximum number of threads to do file system operations on the remote host. Value must be between 1 and 99.
Default: 1.
(--remote-fs-threads={integer})
remote_keep_dir String Move deleted files into the specified directory on the remote host.
(--keep-dir-remote={string})
remote_mount_signature String Verify that the file system is mounted by the existence of this file on the remote host.
(--remote-mount-signature={string})
remote_move_cache_timeout_seconds Integer Delay in seconds before aborting moving a file from remote cache to final destination. -1 for disabled.
Default: -1.
(--remote-move-cache-timeout={integer})
remote_preserve_acls Enum[String] Preserve access control lists on the remote host. If not specified, the default behavior is to use the same storage mode as specified by preserve_acls.
Allowed values: native, metafile, none.
(--remote-preserve-acls={enum})
remote_preserve_xattrs Enum[String] Preserve extended attributes on the remote host. If not specified, the default behavior is to use the same storage mode as specified by preserve_xattrs.
Allowed values: native, metafile, none.
(--remote-preserve-xattrs={enum})
remote_scan_interval_milliseconds Integer Enable periodic scans on the remote host. -1 for disabled. On the command line, the value is a duration, e.g. 1h 30m, 2s500ms, 5:30, 1d 02:00:00.250. A plain number is in seconds. Stored in milliseconds.
Default: -1.
(special:--remote-scan-interval={integer})
remote_scan_threads Integer Number of directory scanning threads on the remote host. Value must be between 1 and 99.
Default: 1.
(--remote-scan-threads={integer})
remote_stat_cache_size Integer Set stat cache size on the remote host. 0 for disabled.
Default: 0.
remove_after_transfer Bool Remove source files after they are successfully synchronized.
Default: false.
(--remove-after-transfer)
reset Bool Clear the snapshot database and rescan the synchronized directories and files to create a fresh snapshot
Default: false.
(--reset)(-x)
resume Hash Partial transfers may exist if communication disruptions caused the underlying ascp processes to terminate early. Note that transfer resumption can only happen if the reset option is disabled. If an async session starts with reset enabled and resume enabled, transfers interrupted during that session will be resumable, but only if async is then restarted with 'reset' disabled.
resume.enabled Bool Enable the possibility of resuming individual file transfers between async sessions.
Default: false.
resume.max_age Integer Sets the age limit in days for temporary files that will be preserved on cleanup (usually at async's start and stop) for potential transfer resume. Temp files older than the given value will be removed regardless of whether they might be resumable.
Default: 5.
(--resume-age-days={integer})
resume.min_size Integer This field specifies the minimum size of files that will be allowed to resume.
Default: 1048576.
(--support-resume={integer})
resume_scan Bool Resume the scan from where the previous execution left off.
Default: false.
(--resume-scan)
scan_dir_rename Bool Enable the detection of renamed directories and files compared to the previous scan, based on matching inodes
Default: false.
(--scan-dir-rename)
scan_file_rename Bool Enable the detection of renamed files compared to the previous scan, based on matching inodes.
Default: false.
(--scan-file-rename)
scan_intensity Enum[String] Scan at the set intensity. vlow minimizes system activity. vhigh maximizes system activity by continuously scanning files without rest.
Allowed values: vlow, low, medium, high, vhigh.
Default: medium.
(--scan-intensity={enum})(-H)
sharing_retry_max Integer Retry synchronizations up to the specified maximum number after a sharing violation.
Default: 3.
(--sharing-retry-max={integer})
store_metadata_records Bool Store the acls or xattrs in the snapshot database.
Default: false.
(--store-metadata-records)
symbolic_links Enum[String] Handle symbolic links with the specified method. Default is skip on windows, copy otherwise.
Allowed values: copy, skip, follow.
(--symbolic-links={enum})(-n)
tags Hash User-defined metadata tags.
(special:--tags64={object})
transfer_threads Array[Hash] Use the specified number of dedicated transfer threads to process files smaller or equal to the specified size
(special:--transfer-threads={array})
transfer_threads[].size Integer Upper limit. -1 for infinity.
Default: -1.
transfer_threads[].threads Integer The number of threads.
Required.
transport Hash
transport.cipher Enum[String] Specify encryption algorithm for file data.
Allowed values: none, aes128, aes192, aes256, aes128cfb, aes192cfb, aes256cfb, aes128gcm, aes192gcm, aes256gcm.
Default: aes128.
(--cipher={enum})(-c)
transport.compression Enum[String] Compress a file before transfer using the specified MODE.
Allowed values: none, zlib.
Default: none.
(--compression={enum})
transport.datagram_size Integer Specify the datagram size (MTU) for FASP. By default it uses the detected path MTU.
(--datagram-size={integer})(-Z)
transport.min_rate Integer Attempt to transfer no slower than the specified rate (in bps).
Default: 0.
(--min-rate={integer})(-m)
transport.rate_policy Enum[String] Defines how ascp will manage the bandwidth.
Allowed values: fair, fixed, high, low.
Default: fair.
(--rate-policy={enum})(-a)
transport.raw_options Array[String] Pass arbitrary arguments to ascp.
(special:--raw-options={array})
transport.read_block_size Integer Use the specified block size (in bytes) for reading. Default is determined by aspera.conf.
(--read-block-size={integer})(-g)
transport.rexmsg_size Integer Use the specified size (in bytes) for a retransmission request. Default is determined by aspera.conf.
(--rexmsg-size={integer})(-X)
transport.target_rate Integer Transfer no faster than the specified rate (in bps). A string with suffix k, m or g is converted to bps by ascli (e.g. 100m).
Default: 10000000.
(--target-rate={integer})(-l)
transport.udp_port Integer Use the specified UDP port for FASP data transfer.
Default: 33001.
(--udp-port={integer})(-O)
transport.write_block_size Integer Use the specified block size (in bytes) for writing. Default is determined by aspera.conf.
(--write-block-size={integer})(-G)
watchd Hash When connection is configured, asperawatchd is used to detect the changes on the source directory.
(special:--watchd={object})
watchd.datastore Enum[String] Specify the type of datastore, none for disabled.
Allowed values: none, redis, scalekv.
Default: none.
watchd.domain String Specify the domain. Default is the current username.
watchd.host String Use the specified host name or address to connect to the datastore.
watchd.port Integer Use the specified port.
Default: 31415.
write_gid String Try to write files as the specified group.
(--write-gid={string})
write_uid String Try to write files as the specified user.
(--write-uid={string})

sync_info: args format🔗

This is the legacy syntax. ascli defines a JSON equivalent to regular async options. It is based on a JSON representation of async command line options. Technically, it allows definition of multiple sync sessions in a single command, but ascli only accepts a single session for consistency with the previous syntax.

This format is selected if the sync_info Hash has either key sessions or instance.

The following parameters are automatically filled from mandatory arguments, and are not allowed:

  • direction
  • local_dir
  • remote_dir

Parameter name is set to a default value if not provided in sync_info.

Sync management and monitoring: admin🔗

The admin command provides subcommands to inspect the state of an Async sync session. Most subcommands read the local snap database (snap.db) directly - no server connection is required. The exception is status, which calls the asyncadmin utility available only on server products.

These commands can also be run from the config plugin:

ascli config sync admin

Prerequisites: Most admin subcommands require the sqlite3 gem:

gem install sqlite3

Usage: All admin subcommands share the same argument structure:

ascli ... sync admin <COMMAND> <FOLDER> [<SYNC_INFO>]
  • <COMMAND> - one of the subcommands listed below.
  • <FOLDER> - path to the local database folder (a folder containing a .private-asp subfolder). By default this is the local synchronized folder; if a separate database folder was configured, specify that path instead.
  • [<SYNC_INFO>] - optional Hash to identify the session. If the folder contains only one session it is selected automatically; otherwise provide @: name=<SESSION_NAME>.

The only exception is find, which takes a plain directory path and lists all <SESSION_NAME> found inside it.

Subcommands:

Command Uses Description
find filesystem Search a directory for existing sync sessions and list their names and paths
status asyncadmin Retrieve the status of a running sync session (requires a server-side product)
meta sqlite3 Display session metadata from sync_snapmeta_table
counters sqlite3 Display synchronization counters from sync_snap_counters_table
file_info sqlite3 List the synchronization state of each file from sync_snapdb_table (state, record ID, path, message)
overview sqlite3 List all tables and their columns in the snap database
query sqlite3 Execute an arbitrary SQL statement against the snap database and return the result rows

The query subcommand requires --sql with the full SQL statement to execute. The sqlite3-based subcommands (meta, counters, file_info) accept --sql optionally to append a SQL fragment (for example, a WHERE or ORDER BY clause):

ascli ... sync admin query <FOLDER> --sql="SELECT * FROM sync_snapdb_table WHERE state=20"
ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20"
ascli ... sync admin file_info <FOLDER> --sql="WHERE state=20 ORDER BY f_meta_path"

Snap database schema: The snap database (snap.db) contains the following tables:

sync_snapmeta_table - one row per session, written at start and updated at stop:

Field Type Description
ascp_sids String Last known FASP (ascp) session IDs.
byte_peerxfer Integer Number of bytes transferred by the peer.
byte_synced Integer Number of bytes successfully synchronized.
byte_xferres Integer Number of bytes for which transfer was resumed.
checksum Enum[String] Checksum algorithm.
Allowed values: sha1, md5, sha1_sparse, md5_sparse, none.
cip_true_val Integer Candidate-in-progress counter true value (internal).
condition Enum[String] Last known session condition.
Allowed values: running, stopped.
dbversion String Database schema version.
direction Enum[String] Sync direction.
Allowed values: push, pull, bidi.
exclude_dirs_older_than Integer Exclude directories with modification time older than this epoch value (0 = disabled).
file_peerxfer Integer Number of files transferred by the peer.
file_synced Integer Number of files successfully synchronized.
file_xferres Integer Number of files for which transfer was resumed.
filters String Active include/exclude filter patterns.
inode_format String Inode format used by the session.
ldir String Local directory path.
max_recid Integer Highest record ID allocated in the snapshot table.
pairname String Sync session name.
path_peerxfer Integer Number of paths transferred by the peer.
path_synced Integer Number of paths successfully synchronized.
path_xferres Integer Number of paths for which transfer was resumed.
rdir String Remote directory path.
rhost String Remote host.
scn_true_val Integer Scan counter true value (internal).
start_time String Session start timestamp.
stop_time String Session stop timestamp (written when session ends).
sync_point Integer Synchronization point sequence number.
sync_uuid String UUID identifying this sync pair.

sync_snap_counters_table - one row, updated live during a session:

Field Type Description
conflict Integer Number of paths in the Conflict state.
error Integer Number of paths in the Error state.
files_conflict Integer Number of files in the Conflict state.
files_error Integer Number of files in the Error state.
files_pending Integer Number of files still pending synchronization.
files_potential_conflict Integer Number of files in the Potential conflict state.
files_syncd Integer Number of files in the Syncd state.
pending Integer Number of paths still pending synchronization.
potential_conflict Integer Number of paths in the Potential conflict state.
syncd Integer Number of paths in the Syncd state.
unused Integer Reserved (unused counter slot).

sync_snapdb_table - one row per tracked file or directory:

Field Type Description
c_cont_csum String Current content — checksum.
c_cont_mtime Integer Current content — modification time (epoch).
c_cont_size Integer Current content — file size in bytes.
c_cont_timestamp Integer Current content — internal timestamp.
c_meta_atime Integer Current metadata — access time (epoch).
c_meta_attrs String Current metadata — file attributes.
c_meta_ctime Integer Current metadata — change time (epoch).
c_meta_gid Integer Current metadata — group ID (Unix).
c_meta_mtime Integer Current metadata — modification time (epoch).
c_meta_object_metadata String Current metadata — object metadata (JSON).
c_meta_ol_legal_hold Integer Current metadata — object lock legal hold flag.
c_meta_ol_mode Integer Current metadata — object lock mode.
c_meta_ol_retain_until Integer Current metadata — object lock retain-until date (epoch).
c_meta_parentrid Integer Current metadata — parent record ID.
c_meta_path String Current metadata — file path as known to the snapshot.
c_meta_type Integer Current metadata — file type (file, directory, …).
c_meta_uid Integer Current metadata — user ID (Unix).
c_meta_xattrs Integer Current metadata — extended attributes (not used).
c_meta_xattrs_cksum String Current metadata — checksum of extended attributes.
candidate_in_prog Integer Set to 1 when this record is currently in the candidate transfer list.
commit_version Integer Local commit version for this entry.
f_cont_csum String Filesystem content — checksum.
f_cont_mtime Integer Filesystem content — modification time (epoch).
f_cont_size Integer Filesystem content — file size in bytes.
f_cont_timestamp Integer Filesystem content — internal timestamp.
f_meta_atime Integer Filesystem metadata — access time (epoch).
f_meta_attrs String Filesystem metadata — file attributes.
f_meta_ctime Integer Filesystem metadata — change time (epoch).
f_meta_gid Integer Filesystem metadata — group ID (Unix).
f_meta_mtime Integer Filesystem metadata — modification time (epoch).
f_meta_object_metadata String Filesystem metadata — object metadata (JSON).
f_meta_ol_legal_hold Integer Filesystem metadata — object lock legal hold flag.
f_meta_ol_mode Integer Filesystem metadata — object lock mode.
f_meta_ol_retain_until Integer Filesystem metadata — object lock retain-until date (epoch).
f_meta_parentrid Integer Filesystem metadata — parent record ID.
f_meta_path String Filesystem metadata — file path as seen on disk (displayed by file_info).
f_meta_type Integer Filesystem metadata — file type.
f_meta_uid Integer Filesystem metadata — user ID (Unix).
f_meta_xattrs Integer Filesystem metadata — extended attributes (not used).
f_meta_xattrs_cksum String Filesystem metadata — checksum of extended attributes.
message String Error or status message associated with this entry.
meta_lastreq_no Integer Sequence number of the last metadata request sent.
meta_lastresp_no Integer Sequence number of the last metadata response received.
nodeid Integer Local filesystem node ID.
p_cont_csum String Pending content — checksum.
p_cont_mtime Integer Pending content — modification time (epoch).
p_cont_size Integer Pending content — file size in bytes.
p_cont_timestamp Integer Pending content — internal timestamp.
pcr_reqno Integer Peer content request sequence number.
peer_commit_version Integer Peer commit version for this entry.
peer_nodeid Integer Peer filesystem node ID.
peer_rid Integer Peer record ID for this entry.
pp_cont_csum String Peer pending content — checksum.
pp_cont_mtime Integer Peer pending content — modification time (epoch).
pp_cont_size Integer Peer pending content — file size in bytes.
pp_cont_timestamp Integer Peer pending content — internal timestamp.
record_id Integer Primary key (auto-increment).
recursive_mtime Integer Most recent modification time across all descendants (directories only).
scn_completed Integer Set to 1 when the scan of this directory entry has completed.
scn_not_visited Integer Set to 1 when the scanner has not yet visited this entry in the current scan pass.
scn_required Integer Set to 1 when a rescan of this entry is required.
state Integer Sync state of the entry.

| Value | Name |
|-------|-------------------|
| 0 | Nil |
| 1–18 | Pending (variant) |
| 19 | Syncd |
| 20 | Error |
| 21 | Conflict |
| 22 | PotentialConflict |
| 23–24 | Pending (variant) |
tobe_deleted Integer Set to 1 when the record is flagged for deletion (garbage collection pending).
version Integer Schema version field (not used).

Hot folder🔗

ascli does not have a standalone hotfolder command. Instead, you create a Hot Folder by combining the upload or download commands with specific transfer-spec parameters and an OS scheduler (like cron or Task Scheduler).

Requirements🔗

ascli may be used as a simple hot folder engine. A hot folder is defined as a tool that:

  • Locally (or remotely) detects new files in a top folder
  • Sends detected files to a remote (respectively, local) repository
  • Only sends new files, and does not re-send already sent files
  • Optionally: sends only files that are not still growing
  • Optionally: after transfer of files, deletes or moves to an archive

In addition: the detection should be made continuously or on specific time/date.

Setting up a hot folder🔗

The general idea is to rely on:

  • Existing ascp features for detection and transfer
  • ascli configuration capabilities and server-side knowledge
  • The OS scheduler for reliability and continuous operation

ascp features🔗

Useful ascp features are available as arguments (see the ascp manual):

  • Sending only new files
  • Option -k 1,2,3 (resume_policy)
  • Remove or move files after transfer:
  • --remove-after-transfer (remove_after_transfer)
  • --move-after-transfer (move_after_transfer)
  • --remove-empty-directories (remove_empty_directories)
  • Send only files not modified since the last X seconds:
  • --exclude-newer-than (exclude_newer_than)
  • --exclude-older-than (exclude_older_than)
  • Top level folder should not be created on destination
  • --src-base (src_base)

Note

ascli takes transfer parameters exclusively as a transfer-spec, with ts option.

Note

Usual native ascp arguments are available as standard transfer-spec parameters, but not special or advanced options.

Tip

Only for the direct transfer agent (not others, like connect or node), native ascp arguments can be provided with parameter ascp_args of option transfer.

Server side and configuration🔗

Virtually any transfer on a repository on a regular basis might emulate a hot folder.

Note

File detection is not based on events (inotify, and so on), but on a simple folder scan on source side.

Tip

Options may be saved in an Option Preset and used with -P.

Scheduling🔗

Once ascli command line arguments are defined, run the command using the OS native scheduler, for example, every minute or every 5 minutes. See Scheduler (and option lock_port).

Example: Upload hot folder🔗

ascli server upload source_hot --to-folder=/Upload/target_hot --lock-port=12345 --ts=@json:'{"remove_after_transfer":true,"remove_empty_directories":true,"exclude_newer_than":-8,"src_base":"source_hot"}'

The local folder (here, relative path: source_hot) is sent (upload) to an Aspera server. Source files are deleted after transfer. Growing files will be sent only once they do not grow anymore (based on an 8-second cool-off period). If a transfer takes more than the execution period, then the subsequent execution is skipped (lock_port) preventing multiple concurrent runs.

Example: Unidirectional synchronization (upload) to server🔗

ascli server upload source_sync --to-folder=/Upload/target_sync --lock-port=12345 --ts=@json:'{"resume_policy":"sparse_csum","exclude_newer_than":-8,"src_base":"source_sync"}'

This can also be used with other folder-based applications: Aspera on Cloud, Shares, Node.

Example: Unidirectional synchronization (download) from Aspera on Cloud Files🔗

ascli aoc files download . --to-folder=. --lock-port=12345 --progress-bar=no --out.level=data --ts=@json:'{"resume_policy":"sparse_csum","target_rate_kbps":50000,"exclude_newer_than":-8,"delete_before_transfer":true}'

Note

Option delete_before_transfer will delete files locally, if they are not present on remote side.

Note

Options progress_bar and --out.level limit output for headless operation (for example, a cron job).

Health check and Nagios🔗

Most plugins provide a health command that checks the health status of the application. Example:

ascli console health
╭────────┬─────────────┬────────────╮
│ status │ component   │ message    │
╞════════╪═════════════╪════════════╡
│ ok     │ console api │ accessible │
╰────────┴─────────────┴────────────╯

Typically, the health check uses the REST API of the application, with the following exception: the server plugin checks health by issuing a transfer to the server (server health transfer).

ascli can be called by Nagios to check the health status of an Aspera server. The output can be made compatible with Nagios with option --format=nagios:

ascli server health transfer --to-folder=/Upload --format=nagios --progress-bar=no
OK - [transfer:ok]

SMTP for email notifications🔗

ascli can send emails. To do so, set up the SMTP configuration with option smtp.

The smtp option is a Hash (Extended Value) with the following fields:

Field Type Description
domain String Email domain of the user. Defaults to the domain part of server.
Example: gmail.com.
from_email String Sender address (used in From: header and as reply-to). Defaults to username when defined.
Example: johnny@example.com.
from_name String Display name of the sender. Defaults to the local-part of from_email.
Example: John Wayne.
password String Password for the above username.
Example: <PASSWORD>.
port Integer Port for the SMTP service.
Defaults: 587 (TLS), 465 (SSL), 25 (plain).
Example: 587.
server String SMTP server address.
Required.
Example: smtp.gmail.com.
ssl Bool Enable TLS/SSL (port 465).
Default: false.
tls Bool Enable STARTTLS (port 587).
Default: true.
username String User to authenticate on the SMTP server. Leave empty for open (unauthenticated) relay.
Example: john@example.com.

Example of configuration🔗

ascli config preset set smtp_google server smtp.google.com
ascli config preset set smtp_google username john@gmail.com
ascli config preset set smtp_google password <PASSWORD>

or

ascli config preset initialize smtp_google @json:'{"server":"smtp.google.com","username":"john@gmail.com","password":"<PASSWORD>"}'

or

ascli config preset update smtp_google --server=smtp.google.com --username=john@gmail.com --password=<PASSWORD>

Set this configuration as global default, for instance:

ascli config preset set cli_default smtp @val:@preset:smtp_google
ascli config preset set default config cli_default

Email templates🔗

Sent emails are built using a template that uses the ERB syntax.

The template is the full SMTP message, including headers.

The following variables are defined by default:

  • from_name
  • from_email
  • to

Other variables are defined depending on context.

Test🔗

Check settings with smtp_settings command. Send test email with email_test.

ascli config smtp_settings --smtp=@preset:smtp_google
ascli config email_test --smtp=@preset:smtp_google --notify-to=sample.dest@example.com

Notifications for transfer status🔗

An e-mail notification can be sent upon transfer success and failure (one email per transfer job, one job being possibly multi session, and possibly after retry).

To activate, use option notify_to.

A default e-mail template is used, but it can be overridden with option notify_template.

The environment provided contains the following additional variables:

  • subject : a default subject including transfer status
  • status : global status of transfer
  • ts : the transfer-spec used for the transfer
  • from_email : email of sender (from smtp configuration)
  • from_name : name of sender (from smtp configuration)
  • to : recipient of the email (from notify_to)

Example of template:

From: <%=from_name%> <<%=from_email%>>
To: <<%=to%>>
Subject: <%=subject%>

Transfer is: <%=status%>

Tool: asession🔗

This gem comes with a second executable tool providing a simplified standardized interface to start a FASP session: asession.

It aims at simplifying the startup of a FASP session from a programmatic standpoint as formatting a transfer-spec is:

  • Common to Aspera Node API (HTTP POST /ops/transfer)
  • Common to Aspera Connect API (browser JavaScript startTransfer)
  • Generated by using any third-party language-specific JSON library

Ideally, IBM will integrate this directly into ascp, making this tool redundant.

Integration with any language is possible, provided that the language can spawn a subprocess, write to its STDIN, read from STDOUT, and generate and parse JSON.

asession expects a single argument: a session specification that contains parameters and a transfer-spec.

If no argument is provided, it assumes a value of: @json:@stdin:, that is, JSON on stdin.

Note

If JSON is the format, specify @json: to tell asession to decode the Hash using JSON syntax.

During execution, it generates all low level events, one per line, in JSON format on stdout.

Top level parameters supported by asession:

Parameter Description
spec The transfer-spec
agent Same parameters as transfer option for agent direct
loglevel Log level of asession
file_list_folder The folder used to store (for garbage collection) generated file lists.
Default: [system tmp folder]/[username]_asession_filelists

Comparison of interfaces🔗

Feature/Tool Transfer Daemon FASPManager ascp asession
status Supported Deprecated Supported Deprecated
language integration Many C/C++
C#/.net
Go
Python
java
Any Any
required additional components to ascp Daemon Library
(+headers)
- Ruby
Aspera gem
startup Daemon API Command line arguments JSON on stdin
(standard APIs:
JSON.generate
Process.spawn)
events Poll Callback Possibility to open management port
and proprietary text syntax
JSON on stdout
platforms Like ascp and transferd Like ascp and lib (if compiled) Any with ascp Any with Ruby and ascp

Simple session🔗

Create a file session.json with:

{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}],"resume_level":"none"}

Then start the session:

asession < session.json

Asynchronous commands and Persistent session🔗

asession also supports asynchronous commands (on the management port). Instead of the traditional text protocol as described in ascp manual, the format for commands is: one single line per command, formatted in JSON, where parameters shall be snake style, for example: LongParameter → long_parameter

This is particularly useful for a persistent session (with the transfer-spec parameter: "keepalive":true)

asession
{"remote_host":"demo.asperasoft.com","ssh_port":33001,"remote_user":"asperaweb","remote_password":"<PASSWORD>","direction":"receive","destination_root":".","keepalive":true,"resume_level":"none"}
{"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
{"type":"DONE"}

(events from FASP are not shown in above example. They would appear after each command)

Help🔗

asession -h
USAGE
    asession
    asession -h|--help
    asession [<session spec extended value>]

    If no argument is provided, default will be used: @json:@stdin
    -h, --help display this message
    <session spec extended value> a dictionary (Hash)
    The value can be either:
       the JSON description itself, e.g. @json:'{"xx":"yy",...}'
       @json:@stdin, if the JSON is provided from stdin
       @json:@file:<path>, if the JSON is provided from a file
    The following keys are recognized in session spec:
       spec : mandatory, contains the transfer spec
       loglevel : modify log level (to stderr)
       agent : modify transfer agent parameters, e.g. ascp_args
       file_list_folder : location of temporary files
       sdk : location of SDK (ascp)
    Asynchronous commands can be provided on STDIN, examples:
       {"type":"START","source":"/aspera-test-dir-tiny/200KB.2"}
       {"type":"START","source":"xx","destination":"yy"}
       {"type":"DONE"}
EXAMPLES
    asession @json:'{"spec":{"remote_host":"demo.asperasoft.com","remote_user":"asperaweb","ssh_port":33001,"remote_password":"<PASSWORD>","direction":"receive","destination_root":"./test.dir","paths":[{"source":"/aspera-test-dir-tiny/200KB.1"}]}}'
    echo '{"spec":{"remote_host":...}}'|asession @json:@stdin:

Ruby Module: Aspera🔗

Main components:

  • Aspera generic classes for REST and OAuth
  • Aspera::Agent::Direct: Starting and monitoring transfers using ascp.
  • Aspera::Cli: ascli.

Working examples can be found in repo: https://github.com/laurent-martin/aspera-api-examples in Ruby examples.