IBM Aspera HST Admin Manual Download Page 102

Page: 102 / 353

| ascp: Transferring from the Command Line with Ascp |

102

Galois/counter mode (GCM). The GCM mode encrypts data faster and increases transfer speeds

compared to the CFB mode, but the server must support and permit it.

Cipher rules

The encryption cipher that you are allowed to use depends on the server configuration and the

version of the client and server:
• When you request a cipher key that is shorter than the cipher key that is configured on the server,

the transfer is automatically upgraded to the server configuration. For example, when the server

setting is AES-192 and you request AES-128, the server enforces AES-192.

• When the server requires GCM, you must use GCM (requires version 3.9.0 or newer) or the

transfer fails.

• When you request GCM and the server is older than 3.8.1 or explicity requires CFB, the transfer

fails.

• When the server setting is "any", you can use any encryption cipher. The only exception is when

the server is 3.8.1 or older and does not support GCM mode; in this case, you cannot request

GCM mode encryption.

• When the server setting is "none", you must use "none". Transfer requests that specify an

encryption cipher are refused by the server.

Cipher Values

Value

Description

Support

aes128

aes192

aes256

Use the GCM or CFB encryption

mode, depending on the server

configuration and version (see cipher

negotiation matrix).

All client and server versions.

aes128cfb

aes192cfb

aes256cfb

Use the CFB encryption mode.

Clients version 3.9.0 and newer, all

server versions.

aes128gcm

aes192gcm

aes256gcm

Use the GCM encryption mode.

Clients and servers version 3.9.0 and

newer.

none

Do not encrypt data in transit. Aspera

strongly recommends against using

this setting.

All client and server versions.

Client-Server Cipher Negotiation

The following table shows which encryption mode is used depending on the server and client

versions and settings:

Server, v3.9.0+
AES-XXX-GCM

Server, v3.9.0+
AES-XXX-CFB

Server, v3.9.0+
AES-XXX

Server, v3.8.1 or

older
AES-XXX

Client, v3.9.0+
AES-XXX-GCM

GCM

server refuses

transfer

GCM

server refuses

transfer

Client, v3.9.0+
AES-XXX-CFB

server refuses

transfer

CFB

Summary of Contents for Aspera HST

Page 1: ...High Speed Transfer Server Admin Guide 3 9 1 PowerLinux Revision 1978 Generated 04 05 2019 10 17...

Page 2: ...iated Transfer 20 Updating the Product License 21 Uninstalling 21 Set up the HST Server Web UI 22 Configuring the Apache Server to Host the HST Server Web UI 22 Configuring your Web UI Settings 25 Cus...

Page 3: ...les 97 ascp Transferring from the Command Line with Ascp 99 Ascp Command Reference 99 Ascp General Examples 114 Ascp File Manipulation Examples 116 Ascp Transfers with Object Storage and HDFS 118 Tran...

Page 4: ...ice Configuration 213 Setting Custom Watch Scan Periods 215 Managing Watch Subscriptions 215 Transferring and Deleting Files with the Aspera Watch Service 216 Aspera Sync 218 Introduction 218 Overview...

Page 5: ...our Nodes 284 Installing SSL Certificates 286 Authentication and Authorization 289 Introduction to Aspera Authentication and Authorization 289 Require Token Authorization Set from the Command Line 290...

Page 6: ...ervices 333 Docroot vs File Restriction 334 Aspera Ecosystem Security Best Practices 335 Securing the Systems that Run Aspera Software 335 Securing the Aspera Applications 338 Securing Content in your...

Page 7: ...cipates in transfers The server can be an on premises installation of HST Server IBM Aspera High Speed Transfer Endpoint which permits one client connection a HST Server installed as part of IBM Asper...

Page 8: ...that storage HST Server can be incorporated into a scalable Aspera data transfer ecosystem that meets your needs Your Aspera server can be monitored and managed by IBM Aspera Console and added as a no...

Page 9: ...r destination for authorized transfers Your server can also take the role of a client and connect to other Aspera servers to initiate transfers The following steps describe how to prepare your system...

Page 10: ...olders to clients when they are added to a specific folder on the server see Introduction to Watch Folders and the Aspera Watch Service on page 159 If you want to enable server based clients to synchr...

Page 11: ...lders Watch Folders Aspera Sync Supported platforms Windows only Windows macOS Linux AIX Solaris Linux on z Systems BSD Isilon Windows macOS Linux AIX Solaris Linux on z Systems BSD Additional license...

Page 12: ...e system change in the Hot Folder is detected On a user specified schedule Immediate as soon as a difference between snapshots is detected Immediate in continuous mode or when using Aspera Sync with a...

Page 13: ...hat supports LE SSH Server Version 7 0 or higher is recommended To use the Node API The line 127 0 0 1 localhost must appear in the hosts file etc hosts For UNIX based nodes SELinux must be set to per...

Page 14: ...d dependencies are installed with your Aspera application by installing the product with a yum install yum nogpgcheck install path_to_installer aspera hsts version rpm On some CentOS 7 and Fedora syst...

Page 15: ...tication to yes To allow password authentication set PasswordAuthentication to yes For example PubkeyAuthentication yes PasswordAuthentication yes c Save the file then reload the SSH service d Restart...

Page 16: ...ee Controlling Bandwidth Usage with Virtual Links Command Line on page 58 Remote Client Machines Typically consumer and business firewalls allow direct outbound connections from client computers on TC...

Page 17: ...ection tab click Show Advanced Settings and enter the SSH port number in the SSH Port TCP field Command line Clients running FASP transfers from the command line can specify the port by using the P 33...

Page 18: ...ure authentication methods add or uncomment PubkeyAuthentication yes and comment out PasswordAuthentication yes PubkeyAuthentication yes PasswordAuthentication yes PasswordAuthentication no Note If yo...

Page 19: ...ku sshd 1496 Failed password for invalid user alex from 1 2 3 4 port 1585 ssh2 Mar 14 23 25 52 sku sshd 1496 Failed password for invalid user alice from 1 2 3 4 port 1585 ssh2 If you identify attacks...

Page 20: ...h_host_key_path 2 Restart the node service to activate your changes Run the following commands to restart asperanoded systemctl restart asperanoded or for Linux systems that use init d service asperan...

Page 21: ...cp T tmp 100MB aspera demo asperasoft com Upload Updating the Product License Update your product license from the command line 1 Open the license file with write permission opt aspera etc aspera lice...

Page 22: ...tions describe how to configure your system s Apache server to host HST Server s web UI The Apache files might be located in different paths or your Apache server could require additional settings dep...

Page 23: ...on to SSH authentication HST Server uses Apache s authentication to authorize web UI access To set up a system user asp1 in this example for Apache authentication run the htpasswd command below Note O...

Page 24: ...Paste the output generated when you ran the enablesecure script as described above BEGIN IBM Aspera High Speed Transfer Server The user account that runs the web server will impersonate the logged in...

Page 25: ...nging it to TCP 33001 as described in Securing your SSH Server The default configuration example above assumes your SSH port is set to TCP 33001 The table below provides descriptions of all web UI con...

Page 26: ...If the minimum version is not installed a message is displayed that indicates the minimum version required and provides a download link This option takes the value in the format of the Connect version...

Page 27: ...wing locations Header opt aspera var webtools aspdir header html Footer opt aspera var webtools aspdir footer html 2 Modify the header and footer then save your changes Testing the Web UI Once your HS...

Page 28: ...n file_system access paths path absolute sandbox name absolute path paths access file_system default CONF To add the settings to aspera conf manually open it from the following directory opt aspera et...

Page 29: ...back HTTP fallback serves as a secondary transfer method when the Internet connectivity required for Aspera FASP transfers UDP port 33001 by default is unavailable When HTTP fallback is enabled and UD...

Page 30: ...Save and close the file d Confirm that aspera conf is formed correctly Validate the aspera conf file using the asuserdata utility opt aspera bin asuserdata v 2 Configure HTTP HTTPS fallback settings R...

Page 31: ...o allow interrupted transfers to resume from the point of interruption true or false true Session Activity Timeout Any value greater than 0 sets the amount of time in seconds that the HTTP fallback se...

Page 32: ...20 recommended This adds or updates the encryption_key value in the authorization section Important After changing your Aspera token settings either in aspera conf or the GUI you must restart asperaht...

Page 33: ...nfigured as Aspera transfer users before clients can browse the server file system or run FASP transfers to and from the server When creating transfer users you can also specify user specific settings...

Page 34: ...user s home directory home Documents 3 For server security Aspera recommends restricting users read write and browse permissions Users are given read write and browse permissions to their docroot by...

Page 35: ...rname absolute docroot asconfigurator x set_user_data user_name username transfer_in_bandwidth_flow_target_rate_default rate asconfigurator x set_user_data user_name username transfer_out_bandwidth_fl...

Page 36: ...is a member of multiple groups the precedence setting can be used to determine priority aspera conf Authorization Configuration on page 40 Connection permissions token key and encryption requirements...

Page 37: ...ealm users user specific settings users groups group Each group tag contains a group s profile name aspgroup name The group name precedence 0 precedence Group precedence authorization authorization Au...

Page 38: ...the Server Public key authentication is an alternative to password authentication providing a more secure authentication method that allows users to avoid entering or storing a password or sending it...

Page 39: ...must have at least one Aspera transfer user a system user account that is configured to authenticate Aspera transfers configured on it If any of the following connection tests fail see Clients Can t...

Page 40: ...t Address ip_address 10 0 0 2 Destination Folder Set the destination path relative to the transfer user s docroot destination dir In this example the files are transferred to the dir folder in the doc...

Page 41: ...yption_key Token Encryption Key filename_hash filename_hash Token Filename Hash life_seconds 86400 life_seconds Token Life seconds token authorization 3 Edit settings as needed Authorization Settings...

Page 42: ...t text string blank Token Encryption Cipher Set the cipher used to generate encrypted transfer tokens aes 128 aes 192 or aes 256 aes 128 Token Encryption Key Set the secret text phrase that is used to...

Page 43: ...Priority Default lock false lock Incoming Priority Lock priority network_rc module delay module Incoming Rate Control Module tcp_friendly false tcp_friendly Incoming TCP Friendly Mode predictor unset...

Page 44: ...tions validation_file_start none validation_file_start Validation File Start validation_file_stop none validation_file_stop Validation File Stop validation_session_start none validation_session_start...

Page 45: ...rver If the client requested minimum rate exceeds network or storage capacity this can decrease transfer performance and cause problems on the target storage positive integer or unlimited unlimited In...

Page 46: ...w bandwidth policy are allowed All others are rejected Incoming Bandwidth Policy Default The default bandwidth policy for incoming transfers Clients can override the default policy if they specify a p...

Page 47: ...l priority setting Use the value 0 to unset this option 1 to allow high priority 2 to enforce normal priority 0 1 or 2 2 Incoming Priority Lock To disallow your clients change the priority set the val...

Page 48: ...ng network congestion When set to unset the client specified predictor is used and if the client does not specify a predictor then none is used For more information see Increasing Transfer Performance...

Page 49: ...ting the minimum rate cap to zero Transfers do not slow below the client s requested minimum rate unless the minimum rate is capped on the server If the client requested minimum rate exceeds network o...

Page 50: ...width policies are allowed Transfers that request fixed bandwidth policy are rejected low Only transfers that use a low bandwidth policy are allowed All others are rejected high fair low or any any Ou...

Page 51: ...o unset this option 1 to allow high priority 2 to enforce normal priority 0 1 or 2 1 Outgoing Priority Default The initial priority setting Use the value 0 to unset this option 1 to allow high priorit...

Page 52: ...rease transfer rate stability and throughput by predicting network congestion When set to unset the client specified predictor is used and if the client does not specify a predictor then none is used...

Page 53: ...and supports two encryption modes cipher feedback mode CFB and Galois counter mode GCM The GCM mode encrypts data faster and increases transfer speeds compared to the CFB mode but the server must supp...

Page 54: ...d is set to true in aspera conf and you use passphrase protected SSH keys you must use keys generated by running ssh keygen in a FIPS enabled system or convert existing keys to a FIPS compatible forma...

Page 55: ...ession below the input value The default of 0 will cause the Aspera sender to use its default internal buffer size which may be different for different operating systems positive integer 0 Minimum Soc...

Page 56: ...For threshold validation the file transfer might complete before the file threshold validation response comes back because ascp doesn t pause file transfers during file threshold validation therefore...

Page 57: ...o Lua Action Script must be defined if any of the following values are set to lua_script Run at File Start Run at File Stop Run at Session Start Run at Session Stop Run when Crossing File Threshold If...

Page 58: ...users create a Vlink with a 10 Mbps capacity and assign it to outgoing transfers for those three users If the three users are running download sessions that already use 10 Mbps and another download is...

Page 59: ...era conf CONF version 2 trunks trunk id 108 id Vlink ID name 50Mbps cap name Vlink Name capacity schedule format ranges 50000 schedule Capacity capacity on true on On trunk trunks CONF The capacity of...

Page 60: ...2 default transfer out bandwidth aggregate trunk_id 108 trunk_id Vlink 108 for the default outgoing sessions aggregate bandwidth out in in transfer default aaa realms realm users user name aspera_user...

Page 61: ...x set_trunk_data id 108 trunk_capacity 88000 trunk_on true asconfigurator x set_trunk_data id 109 trunk_capacity 99000 trunk_on true asconfigurator x set_node_data transfer_in_bandwidth_aggregate_trun...

Page 62: ...fference method bezier A quadratic Bezier extrapolation ets An error trend seasonality model Based on internal testing fd31 is considered the most effective and robust but other RTT predictors might p...

Page 63: ...a docroot For more information see Docroot vs File Restriction on page 334 Configuration methods These instructions describe how to manually modify aspera conf You can also add and edit these paramet...

Page 64: ...file_suffix partial partial_file_suffix Partial File Suffix file_checksum any file_checksum File Checksum Method file_system 3 Edit settings as needed File System Settings Reference Field Description...

Page 65: ...ictions do not start with the user can access any file that matches any one of the no restrictions Format examples For a specific folder For the drive root For ICOS S3 storage s3 my_vault To exclude a...

Page 66: ...teger 0 Number of Dir Scanning Threads Set the number of threads the Aspera sender uses to scan directory contents It takes effect on both client and server when acting as a sender The default of zero...

Page 67: ...ed for resuming incomplete transfers Each data file in progress will have a corresponding metadata file with the same name plus the resume suffix specified by the receiver Metadata files in the source...

Page 68: ...calculating job size before transferring Set to no to disable calculating job size before transferring Set to any to follow client configurations yes no or any any Convert Restricted Windows Characte...

Page 69: ...to manage the asperacentral database Configuration methods These instructions describe how to manually modify aspera conf You can also add and edit these parameters using asconfigurator commands For m...

Page 70: ...torage Path Valid system path If the application is installed in the default location then the path is the following Maximum Age seconds Maximum allowable age in seconds of data to be retained in the...

Page 71: ...listed in aspera conf Filtering is a process of exclusion and include rules override exclude rules that follow them Include rules cannot add back files that are excluded by a preceding exclude rule I...

Page 72: ...attern abc f matches abcdef but not abcdefg For details on using wildcards and special characters to build rule patterns see Using Filters to Include and Exclude Files on page 126 Set Rules Filter rul...

Page 73: ...or user asp1 asconfigurator x set_user_data user_name asp1 file_filters abc wxy tuv abc def Results in aspera conf aaa realms realm users user name asp1 name file_system filters filter abc wxy tuv fil...

Page 74: ...ansfers FASP and HTTP fallback transfers Requirements If the following requirements are not met then the server can have both encrypted and unencrypted content This can cause file corruption on the se...

Page 75: ...d Connect Server or HST Endpoint formerly Point to Point Client version 3 4 2 or higher The transfer must be encrypted Encryption is enabled by default The user on the destination can calculate a chec...

Page 76: ...t source path at the sender Note File manifests can be stored only locally Thus if you are using S3 or other non local storage you must specify a local manifest path Enabling checksum reporting by edi...

Page 77: ...ieve the checksum that was calculated by Aspera as the file was transferred If you specified a file manifest and file manifest path as part of an ascp transfer or pre post processing script the checks...

Page 78: ...port user specific logging settings If the client specifies a log directory on the server using R remote_log_dir or the location and size of the local log directory using L local_log_dir size then the...

Page 79: ...The full path to the logging directory Applies only to ascp transfers log_size The size of the log file in MB at which it is rotated the oldest information is overwritten by the newest information Def...

Page 80: ...transfer file validation is run as soon as the client uploads a to HST Server The transfer is reported as complete and then the validation is run The validation script uses the Aspera Reliable Query A...

Page 81: ...max_result sets a batch size for how many files are collected for validation by each POST request and cannot exceed 1000 The POST request retrieves the files that are to_be_validated updates their st...

Page 82: ...session_uuid session_uuid file_id file_id status error error_code error_number error_description error_string https server_name 9092 services rest transfers v1 files For example the body of a PUT req...

Page 83: ...ntly be executed by an external product that integrates with an Aspera product Inline file validation is a feature that enables file content to be validated while the file is in transit as well as whe...

Page 84: ...s IP address and port and the servlet name URL handler found in web xml This adds the path to the transfer section of aspera conf For example transfer validation_uri http 127 0 0 1 8080 SimpleValidat...

Page 85: ...2 defining values in aspera conf For more information on the output of your inline validation see Inline File Validation with URI on page 85 or Inline File Validation with Lua Script on page 87 Inline...

Page 86: ...t javax servlet http HttpServlet import javax servlet http HttpServletRequest import javax servlet http HttpServletResponse import java io BufferedReader import java io IOException WebServlet name Sim...

Page 87: ...cript defined in aspera conf The parameters for Lua calls are passed to Lua scripts by using the array env_table The following is an example request body env_table startstop running env_table xfer_id...

Page 88: ...tring AES128 ANY or NONE cookie The cookie sent to the client system String manifest_file Path to manifest file which contains a list of transferred files The command for this in ascp is file manifest...

Page 89: ...S_IFIFO S_IFSOCK S_IFLNK Block stream Custom Unknown stat_data mode format Windows format Linux format stat_data mode filemode format based on mode format above stat_data uid uid stat_data gid gid st...

Page 90: ...ransfer events Session start Session end Start of each individual file transfer in the session End of each individual file transfer in the session The aspera prepost script can also execute additional...

Page 91: ...an be written directly into the script file aspera prepost For example to add the custom script script1 pl to your pre post script insert the following line into aspera prepost perl script1 pl Pre Pos...

Page 92: ...escription Values Example FILE1 The first file string FILE1 first file FILE2 The second file string FILE2 second file FILECOUNT The number of files positive integer FILECOUNT 5 FILELAST The last file...

Page 93: ...itive integer STARTBYTE 100000 Pre Post Script Examples The following pre processing and post processing script examples demonstrate how Aspera prepost environment variables are used to achieve differ...

Page 94: ...y must be cached bin bash TARGET aspera 10 10 10 10 tmp RATE 10m export ASPERA_SCP_PASS aspera if TYPE File then if STARTSTOP Stop then if STATE success then if DIRECTION recv then logger plocal2 info...

Page 95: ...repare the email notification configuration template Open the aspera conf file opt aspera etc aspera conf Locate or create the section EMAILNOTIF EMAILNOTIF CONF version 2 EMAILNOTIF MAILLISTS mylist...

Page 96: ...in FILTER FILTER defines email notification conditional filters When the conditions are met a customized email is sent to the indicated mailing list Multiple filters are allowed The values in the filt...

Page 97: ...SENDONSESSION yes yes no SENDONSTOP yes SENDONFILE Send email for each file within a session yes no SENDONFILE yes Email Notification Examples Use the following examples to craft your own email notif...

Page 98: ...e is sent to mediaGroup When a regular transfer occurs files are sent to upload a different notification is sent to mediaLead and adminGroup EMAILNOTIF MAILLISTS mediaGroup johndoe companyemail com ja...

Page 99: ...e src_host source1 source2 username dest_host dest_path username The username of the Aspera transfer user can be specified as part of the source or destination whichever is the remote server It can al...

Page 100: ...place each backslash in the UNC path with a forward slash For example if the UNC path is 192 168 0 10 temp change it to 192 168 0 10 temp This format can be used with any client side operating system...

Page 101: ...01 0 4137 9e50 201b 63d3 ba92 da path or host fe80 21b 21ff fe1c 5072 eth1 range_start range_end Transfer only part of a file range_start is the first byte to send and range_end is the last If either...

Page 102: ...does not support GCM mode in this case you cannot request GCM mode encryption When the server setting is none you must use none Transfer requests that specify an encryption cipher are refused by the s...

Page 103: ...e transfer delete any files that exist at the destination but not also at the source The source and destination arguments must be directories that have matching names Do not use with multiple sources...

Page 104: ...t files when receiving for client side encryption at rest EAR Encrypted files have the file extension aspera env This option requires the encryption decryption passphrase to be set with the environmen...

Page 105: ...ch source is specified on a separate line with its destination on the line following it Specify destinations relative to the transfer user s docroot Even if a destination is specified as an absolute p...

Page 106: ...ich might vary by operating system The sending server never uses the read_block_size set in the client s aspera conf h help Display the help text host hostname Transfer to the specified host name or a...

Page 107: ...d the source files remain in their original location To preserve portions of the file path above the transferred file or directory use this option with src base For an example see Ascp File Manipulati...

Page 108: ...Rules found in aspera conf are applied before any E and N rules specified on the command line O fasp_port Use the specified UDP port for FASP transfers Default 33001 overwrite never always diff diff...

Page 109: ...rate to fully utilize the available bandwidth up to the maximum rate When congestion occurs bandwidth is shared fairly by transferring at an even rate The fair policy requires maximum target and mini...

Page 110: ...Preserve the group information gid or owner information uid of the transferred files These options require the transfer user to be authenticated as a superuser preserve modification time Set the modi...

Page 111: ...der for the target side of a pull Ascp with mode recv to apply the ACLs remote preserve xattrs native metafile none Like preserve xattrs but used when attributes are stored in a different format on th...

Page 112: ...urce path includes an embedded passphrase the prefix must also include the embedded passphrase otherwise it will not match For examples see Ascp File Manipulation Examples on page 116 symbolic links f...

Page 113: ...ng a version of ascp that is older than 3 3 in which case the client setting is used If the pre 3 3 client does not set Z the datagram size is the discovered MTU and the server logs the message LOG Pe...

Page 114: ...00m O 42000 local dir files user 10 0 0 2 remote dir Public key authentication Transfer with public key authentication using the key file home dir ssh aspera_user_1 key local dir files ascp l 10m i ss...

Page 115: ...ra env from the server 10 0 0 2 and decrypt while transferring export ASPERA_SCP_FILEPASS secRet ascp l 10m file crypt decrypt root 10 0 0 2 remote dir file aspera env local dir Decrypt a downloaded e...

Page 116: ...tents to a new directory by using the d option Upload the data directory to the server and if it doesn t already exist create the new folder storage2 to contain it resulting in storage2 data at the de...

Page 117: ...ot copy srcdir to the archive directory Archive on the server ascp move after transfer Archive Pat 10 0 0 1 srcdir C Users Pat Move the source file on the client after it is uploaded to the server and...

Page 118: ...l Examples on page 114 You are prompted for the transfer user s password when you run an ascp command unless you set the ASPERA_SCP_PASS environment variable or use SSH key authorization With No Docro...

Page 119: ...e Aspera recommends running ascp transfers with Azure Data Lake Storage with a docroot configured Upload syntax ascp options mode send user username host server_address source_files azu storage_accoun...

Page 120: ...ver_address s3 access_id secret_key accessor_endpoint vault_na source_files destination_path Download example ascp mode send user bear host s3 asperasoft com s3 3ITI3OIUFEH233 KrcEW AIuwQ 38 123 76 24...

Page 121: ...O port_1 multi session threashold threshold tags aspera xfer_id transfer_id source_path hostname destination_path ascp C nid_2 ncount l max_rate O port_2 multi session threashold threshold tags aspera...

Page 122: ...n the following asconfigurator x set_node_data transfer_multi_session_threshold_default threshold Multi Session Transfer Example The following example shows a multi session transfer on a dual core sys...

Page 123: ...ite diff and overwrite diff older is undefined Single file Transfer Examples Upload 1025 bytes of data from the client stdin to remote dir on the server at 10 0 0 2 Save the data as the file newfile T...

Page 124: ...rectories are not allowed Only overwrite always or overwrite never are supported with stdio tar The behavior of overwrite diff and overwrite diff older is undefined Offsets are only supported if the d...

Page 125: ...0m mode recv keepalive M 12345 user username host 10 0 0 2 stdio tar Send the following in through management port 12345 FASPMGR 2 Type START Source tmp myfile1 Destination mynewfile1 FASPMGR 2 Type S...

Page 126: ...ing rules are configured in aspera conf they are applied before the rules on the command line Filtering is a process of exclusion and N rules override E rules that follow them N cannot add back files...

Page 127: ...e files are evaluated Example Consider the following command ascp N file2 E file 0 9 images icons user1 examplehost tmp Where images icons is the source If images icons contains file1 file2 and fileA...

Page 128: ...ern Matches directories only With N no files under matched directories or their subdirectories are included in the transfer All subdirectories are still included although their files will not be inclu...

Page 129: ...the target is the Upload directory At the prompt enter the password demoaspera 3 Create a destination directory on your computer for example tmp dest 4 Download your files from the demo server to tmp...

Page 130: ...abc wxy def AAA abc wxy tuv def AAA abc xyz def wxy AAA wxyfile AAA wxy xyx AAA wxy xyxfile 3 Include directories and files that start with wxy if they fall directly under AAA N wxy E AAA Results AAA...

Page 131: ...AA wxy xyx AAA wxy xyxfile AAA abc def AAA abc def AAA abc wxy def 6 Exclude directories and files starting with wxy but only those found at a specific location in the tree E AAA abc wxy Results AAA a...

Page 132: ...c links Copy Client only Copy only the symbolic link If a file with the same name exists at the destination the symbolic link does not replace the file Copy force Client only Copy only the symbolic li...

Page 133: ...asconfigurator x set_user_data user_name username symbolic_links value For more information see aspera conf File System Configuration on page 63 Client Configuration To specify symbolic link handling...

Page 134: ...ne use the option i private_key_file For example ascp T l 10M m 1M i ssh id_rsa myfile txt jane 10 0 0 2 space In this example you are connecting to the server 10 0 0 2 directory space with the user a...

Page 135: ...hen set to text a text file is generated that lists all files in each transfer session file_manifest_path file_manifest_path path The location where manifest files are written The location can be an a...

Page 136: ...ra and find that it is corrupted you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source...

Page 137: ...an encryption password and the files are uploaded to the server with a aspera env extension Anyone downloading these aspera env files must have the password to decrypt them and decryption can occur a...

Page 138: ...t ASPERA_SCP_FILEPASS password opt aspera bin asprotect o file1 aspera env file1 To download client side encrypted files without decrypting them immediately run the transfer without decryption enabled...

Page 139: ...list filepath file manifest none text file manifest path directory file manifest inprogress suffix suffix file pair list filepath G write_size g read_size h help h help i private_key_file_path i priv...

Page 140: ...file owner gid preserve file owner gid preserve file owner uid preserve file owner uid preserve modification time preserve source access time preserve xattrs mode proxy proxy_url q q R remote_log_dir...

Page 141: ...s not exist With ascp4 you must specify d otherwise all the files in the file list are written to a single file i SSH key authentication With ascp the argument for i can be just the file name of the p...

Page 142: ...te regardless of network or storage capacity This can decrease transfer performance and cause problems on the target storage Aspera discourages using the fixed policy except in specific contexts such...

Page 143: ...t On Windows the only option is skip Symbolic link handling also depends on the server configuration and the transfer direction For more information see Symbolic Link Handling on page 132 5 What are m...

Page 144: ...ng For more information see the IBM Aspera Streaming for Video User Guide Required Configuration for Multicast to Multicast Streaming The transfer user who authenticates the data multicast stream tran...

Page 145: ...e specified If a destination path is a URI no docroot upload or local docroot download can be specified The special schemes stdio and stdio tar are supported only on the client side They cannot be use...

Page 146: ...enticate to a URI destination Ascp 4 Options A version Display version and license information c aes128 aes192 aes256 none Encrypt in transit file data using the specified cipher This option overrides...

Page 147: ...age 126 Note When filtering rules are found in aspera conf they are applied before rules given on the command line E and N exclude newer than mtime exclude older than mtime Exclude files but not direc...

Page 148: ...as overwrite diff compare size resume k 2 Compare sparse checksum and resume if they match same as overwrite diff compare md5 sparse resume k 3 Compare full checksum and resume if they match same as o...

Page 149: ...iles at the destination with source files of the same name based on the method Default always Use with compare and resume method can be the following always Always overwrite the file never Never overw...

Page 150: ...lize the available bandwidth up to the maximum rate When congestion occurs bandwidth is shared fairly by transferring at an even rate This option requires maximum target and minimum transfer rates l a...

Page 151: ...fix must specify the URI in the same manner as the source paths For example if a source path includes an embedded passphrase the prefix must also include the embedded passphrase otherwise it will not...

Page 152: ...3 client does not set Z the datagram size is the discovered MTU and the server logs the message LOG Peer client doesn t support alternative datagram size Ascp 4 Transfers with Object Storage Files tha...

Page 153: ...eads and eight read threads on the client and eight meta threads and 16 write threads on the server ascp4 L tmp logs R tmp logs l1g scan threads 2 read threads 8 write threads 16 meta threads 8 data 1...

Page 154: ...tbatch 0 1 Enable packet batching in read write Default 1 maxsize N Set the maximum stream length Default unlimited maxtime N Set the maximum stream duration in seconds Default unlimited maxidle N Set...

Page 155: ...mmended Rate Settings for Video Streams ascp4 Option Description Recommendation m Minimum rate Take the encoding rate of the transport stream and add 1 Mbps l Target rate Take the minimum rate and add...

Page 156: ...one read threads 1 write threads 1 udp 233 3 3 3 3000 loopback 1 ttl 2 udp localhost 3000 Read a TCP stream from 192 168 10 10 port 2000 and send it to 10 10 0 51 On 10 10 0 51 write the stream to loc...

Page 157: ...back 0 ascp4 L opt test local 03 R opt test remote 03 DD m 12m l 15m mode send host 10 132 117 2 user root read threads 1 write threads 1 compression none udp 233 33 3 3 3001 sndbufsz 100MB ifaddr 10...

Page 158: ...ccess Run the following command to unset a docroot and set a file restriction asconfigurator x set_user_data user_name username absolute AS_NULL file_restriction restriction The restriction can be set...

Page 159: ...uted sources On file systems that have file system notifications changes in source file systems new files and directories deleted items and renames are detected immediately eliminating the need to sca...

Page 160: ...nfiguration 1 This is the simplest and most common configuration of Watch Folder services Use an account that has read permissions for all your files and follow the instructions in Creating a Push Wat...

Page 161: ...t with rund watch watchd and watchfolderd opt aspera bin asuserdata a For more information on configuring see Watch Service Configuration on page 213 Watch Folder Service Configuration on page 174 Con...

Page 162: ...asperawatchfolderd as described in Choosing User Accounts to Run Watch Folder Services on page 160 For more information see Starting Aspera Watch Services and Creating Watches on page 211 and Creating...

Page 163: ...ble d109d1bd 7db7 409f bb16 ca6ff9abb5f4 asrun send code 0 null Enable a Service Enabling a stopped service starts the service This command can be used to restart a service that stops due to an error...

Page 164: ...hat use init d service asperarund status Aspera Run Server asperarund RUNNING 2 Select or create a user account to run your services Watch Folder services must be run under a user with access to every...

Page 165: ...Watch Service Configuration on page 213 and Watch Folder Service Configuration on page 174 Your system is now ready for Watch Folders To create a push Watch Folder see Creating a Push Watch Folder wit...

Page 166: ...torage IBM Aspera Shares endpoints must have version Shares version 1 9 11 with the Watch Folder patch or a later version To create a push Watch Folder 1 Prepare your computer as described in Getting...

Page 167: ...rce_directory target path target_directory location type REMOTE host hostname port port authentication type authentication_mode user username pass password keypath key_file watchd scan_period scan_per...

Page 168: ...an access key ID and secret Sample JSON syntax for each authentication type is provided following this table NODE_BASIC user The username for authentication Required Depending on the type of authenti...

Page 169: ...file is the path to the Watch Folder configuration file If you do not know the daemon retrieve a list of running daemons by running the following command opt aspera bin aswatchfolderadmin query daemon...

Page 170: ...directory are re transferred Restrictions on all Watch Folders Only local to remote push and remote to local pull configurations are supported Remote to remote and local to local are not supported Gro...

Page 171: ...send l The output is similar to the following in this example the user is root asrun send code 0 services id d109d1bd 7db7 409f bb16 ca6ff9abb5f4 configuration enabled true run_as pass user root type...

Page 172: ...e Set type to REMOTE for the remote server type REMOTE is assumed if host is specified REMOTE host The host IP address DNS hostname or URL of the remote file system Required The host can be specified...

Page 173: ...age NFS Solaris AIX and Isilon file system scans triggered by the scan period are used to detect file changes In this case set the scan period to frequently scan for changes On operating systems that...

Page 174: ...folders daemon_name For example opt aspera bin aswatchfolderadmin query folders root aswatchfolderadmin query folders Found a single watchfolder b394d0ee 1cda 4f0d b785 efdc6496c585 7 Test your Watch...

Page 175: ...les across all drops When this number is exceeded drops are purged until the file count is less than the specified number 9223372036854775807 watchfolderd_raw_options raw_options Enable the use of new...

Page 176: ..._period 10s meta version 0 name aspera_watchfolder drop detection_strategy COOL_OFF_ONLY cool_off 5s post_processing source type TRANSFER_NONE archive_dir watchfolder_sessions UUID _ DATETIME filters...

Page 177: ...te_blk_size datagram_size rexmsg_size cipher AES128 overwrite DIFF resume NONE preserve_uid false preserve_gid false preserve_time false preserve_creation_time false preserve_modification_time false p...

Page 178: ...pe REMOTE host host port port authentication type SSH NODE_BASIC user username pass password keypath key_file fingerprint ssh_fingerprint target path path id watchfolder_id cool_off 30s snapshot_creat...

Page 179: ...dpoints enter 443 If authentication type is SSH then default is the value for tcp_port in the transport section default 22 If authentication type is NODE_BASIC then default is 9092 authentication type...

Page 180: ...nsferred in the same transfer session post processed together and reported as a unit Watch Folders uses asperawatchd to detect file system modifications and continuously creates snapshots to compute t...

Page 181: ...and reported as a unit drop detection_strategy COOL_OFF_ONLY cool_off 5s Field Description Default detection_strategy The strategy that Watch Folders uses to create drops when new files are added to...

Page 182: ...NSFER_ARCHIVE Files in the source directory are moved to a final archive after successful transfer This option is not supported for sources in object storage TRANSFER_DELETE Files in the source direct...

Page 183: ...NCLUDE and EXCLUDE Note An include rule must be followed by at least one exclude rule otherwise all files are transferred because none are excluded To exclude all files that do not match the include r...

Page 184: ...e syntax as in the filters object N A The transport object Use to configure authentication to the remote host transport host 198 51 100 22 user aspx2 pass XF324cd28 token fiewle535etn23TEIW234n5sEWTns...

Page 185: ...rver on page 16 Configuring Transfer Server Authentication N A tags Specify custom metadata in JSON format The tags object is passed directly to the ascp session For more information on writing custom...

Page 186: ...min_rate 0B target_rate 10M tcp_port 22 udp_port 33001 read_blk_size write_blk_size datagram_size rexmsg_size cipher AES128 overwrite DIFF resume NONE preserve_uid false preserve_gid false preserve_t...

Page 187: ...ata in transit Aspera supports three sizes of AES cipher keys 128 192 and 256 bits and supports two encryption modes cipher feedback mode CFB and Galois counter mode GCM The GCM mode encrypts data fas...

Page 188: ...and servers version 3 9 0 and newer NONE Do not encrypt data in transit Aspera strongly recommends against using this setting All client and server versions Client Server Cipher Negotiation The follow...

Page 189: ...he modification time of the destination file to that of the source false preserve_access_time Set the access time of the destination to that of the source The destination file has the access time of t...

Page 190: ...Folder If a file does not match the growing file filter it is transferred by Ascp Note Growing files are only supported for local sources push Watch Folders and must be authenticated by a transfer us...

Page 191: ...gram size MTU for FASP The detected path MTU cipher The encryption cipher that is used to encrypt streamed data in transit either NONE and AES128 AES128 completion_timeout How long to wait before the...

Page 192: ...anges in the source directory Lower scan periods detect changes faster but can result in greater resource consumption particularly for object storage Note The value for scan period cannot be empty oth...

Page 193: ...Update a Watch Folder s Configuration To update a Watch Folder configuration retrieve the Watch Folder s configuration make the desired changes and then save the configuration as a JSON file You cann...

Page 194: ...max_user_watches 524288 etc sysctl conf 2 Increase the maximum number of inotify instances which correspond to the number of allowed Watch Services instances Retrieve the current value by running the...

Page 195: ...lder source is in object storage IBM Aspera Shares endpoints must have version Shares version 1 9 11 with the Watch Folder patch or a later version To create a push Watch Folder with the API 1 Prepare...

Page 196: ...the next step 5 Confirm that the services are running For each service run the following command curl ki u node_username node_password X GET https localhost 9092 rund services service_id The state is...

Page 197: ...s Zone IDs for example eth0 can be appended to the IPv6 address N A port The port to use for authentication to the remote file system By default if the authentication type is SSH then the SSH port for...

Page 198: ...Windows macOS asperawatchd uses the file notifications as the primary means for detecting changes and the scan period serves as a backup In this case the default value of 30 minutes is usually accepta...

Page 199: ...Growing files are only supported for local sources push Watch Folders and must be authenticated by a transfer user password or SSH key file The transfer user cannot be restricted to aspshell and the...

Page 200: ..._user root admin impersonation 4 Create a Watch Service on the remote server This approach requires that you have node credentials for the remote server a Create a JSON configuration file for the remo...

Page 201: ...rs and growing file handling A basic pull Watch Folder configuration has the following syntax source path source_directory location type REMOTE host ip_address port port authentication type authentica...

Page 202: ...or authentication depending on the type of authentication N A target path The target directory on the local computer relative to the transfer user s docroot N A watchd identifier The daemon associated...

Page 203: ...F324cd28 H X aspera WF version 2017_10_23 X POST d watchfolder_conf json https 198 51 100 22 9092 v3 watchfolders id b394d0ee 1cda 4f0d b785 efdc6496c585 8 Verify that the Watch Folder is running curl...

Page 204: ...a configuration option that was not set Errors with ascp transfers are displayed similarly in the transport section curl ks user watchfolder_admin XF324cd28 H X aspera WF version 2017_10_23 X GET htt...

Page 205: ...rl k user node_api_user node_api_password H X aspera WF version 2017_10_23 X GET https host node_api_port v3 watchfolders 2 Get the ID of the failed drop curl k user node_api_user node_api_password H...

Page 206: ...aspera WF version 2017_10_23 is required when submitting POST PUT and GET requests to v3 watchfolders on servers that are version 3 8 0 or newer This enables Watch Folders to parse the JSON source an...

Page 207: ...essfully deleted Configuring Custom Watch Folder Permissions Policies By default users are not allowed to perform any Watch Folders related actions unless they are configured with admin ACLs If you do...

Page 208: ...te delete and view policies and assign users to policies These actions do not require that you specify a value for resources To allow all permissions use PERM_ PERM_CREATE_POLICY PERM_DELETE_POLICY PE...

Page 209: ...IST_RESOURCES resources arn watchfolder wfd Assigning Node API Users to Policies Assign a user to one or more policies by running the following command curl k user node_api_user node_api_password X PU...

Page 210: ...access to the source directory specified in the JSON configuration file You might have specified a destination that is not permitted by the docroot or restriction of the user running asperawatchfolder...

Page 211: ...folder drive root file c Amazon S3 and IBM Cloud Object Storage S3 s3 Azure azu Azure Files azure files Azure Data Lake Storage adl Alibaba Cloud oss Google Cloud gs HDFS hdfs With a docroot or restri...

Page 212: ...bject storage to which the user has access Users can create Watch Folders and Watch services on files or objects only within their docroot or restriction Note Users can have a docroot or restriction b...

Page 213: ...hed by the Aspera Watch Service To create a watch users subscribe to a Watch Service and specify the path to watch run the following command where daemon is the username used to start the asperawatchd...

Page 214: ...a conf setting Description Default watch_log_dir log_dir Log to the specified directory This setting applies to both the Watch Service and Watch Folders services The Aspera logging file Log Files on p...

Page 215: ...eriod of an existing subscription Set the Default Scan Period When Upgrading from 3 7 4 or earlier to 3 8 0 or later To update the default scan period that is applied during the migration run the foll...

Page 216: ...ing changes and the scan period serves as a backup In this case the default value of 30 minutes is usually acceptable and no change is necessary To never scan and rely entirely on file notifications s...

Page 217: ...scription ID later 2 Create a snapshot opt aspera bin aswatchadmin create snapshot daemon subscription_id If you do not have the subscription ID run the following command opt aspera bin aswatchadmin q...

Page 218: ...hat contain ASCII characters such as or are not deleted and an error is logged CAUTION asdelete follows symbolic links which can result in files being deleted that are not within the target directory...

Page 219: ...irectional synchronization Aspera Sync runs with a bi directional option For a multi directional synchronization one session is run for each peer to remain sync Any topology that has an acyclic graph...

Page 220: ...nce on page 234 This mode should be used for one_time operations or for periodic scheduled synchronizations where file systems do not support event based change notification For the latter async can b...

Page 221: ...session one async process execution for each remote peer Any number of async processes can be run concurrently and any number of peers can be synchronized concurrently however a downstream peer cannot...

Page 222: ...leted or the change occurs on both endpoints concurrently such that the newer version cannot be reliably determined Aspera Sync reports such conflicts and does not modify either file system leaving th...

Page 223: ...ame async_db_dir db_dir This setting overrides the remote database directory specified by the client with the B option Note If the transfer user s docroot is a URL such as file then async_db_dir must...

Page 224: ...c Value has the syntax sqlite lock_style storage_style Default undefined lock_style Specify how async interfaces with the operating system Values depend on operating system Unix based systems have the...

Page 225: ...te_grant_mask Specify the mode for newly created directories if directory_create_mode is not specified If specified directory modes are set to their original modes plus the grant mask values This opti...

Page 226: ...GUI shows transfers associated with a Aspera Sync job in which the remote user aspera is pushing files to the server folder for Project X You can configure the server and client reporting to the Aspe...

Page 227: ...c link If a file with the same name exists at the destination the symbolic link does not replace the file Copy force Client only Copy only the symbolic link If a file with the same name exists at the...

Page 228: ...s a database snap db that is stored on both the local client computer and the remote server computer The database records the state of the file system at the end of the last async session and the next...

Page 229: ...sp ex2 snap db On the remote computer server opt aspera var private asp ex2 snap db storage users ex2 for transfer cache Changing Synchronization Direction Between Runs of the Same Session Changing di...

Page 230: ...ze with AWS S3 storage see Synchronizing with AWS S3 Storage on page 253 1 Confirm that both endpoints have Aspera Sync enabled licenses and that the remote endpoint is running an Aspera transfer serv...

Page 231: ..._TOKEN or in the command line using the W token_string or token token_string option For example use i and specify the path to Morgan s SSH private key in their home folder async L C Users Morgan Asper...

Page 232: ...spera Sync is push or bidirectional use local mount signature If the remote endpoint is on a NFS or CIFS mount and the Aspera Sync is pull or bidirectional use remote mount signature 11 Specify the lo...

Page 233: ...1 data R morgan async log B morgan async db K bidi t Note When synchronizing between Unix like operating systems you can also preserve the user IDs uid and group IDs gid from the source to the destina...

Page 234: ...numeric characters plus _ and characters Note If your remote host is an Aspera cluster ensure that your session name is unique by naming the session with a descriptive string followed by the UUID of t...

Page 235: ...dir are synchronized with newer versions of files and directories overwriting older versions in either ldir or rdir by default Using continuous mode C Continuous mode is supported only when the file s...

Page 236: ...a directory s modification time has not changed compared to the Aspera Sync database async in non continuous mode skips scanning the directory This option makes scanning static directory structures fa...

Page 237: ...but the server must support and permit it Cipher rules The encryption cipher that you are allowed to use depends on the server configuration and the version of the client and server When you request...

Page 238: ...Server v3 9 0 AES XXX Server v3 8 1 or older AES XXX Client v3 9 0 AES XXX GCM GCM server refuses transfer GCM server refuses transfer Client v3 9 0 AES XXX CFB server refuses transfer CFB CFB CFB Cl...

Page 239: ...nd the target files have matching inodes This option is supported only between Unix based platforms If dedup inode is used in a continuous sync Aspera recommends using the scan interval option copy Af...

Page 240: ...ize Use the specified block size for writing size is an integer with units of K M or bytes Default 64 MB g size read block size size Set block size for reading size is an integer with units of K M or...

Page 241: ...type type can be sha1 md5 sha1 sparse md5 sparse or none A value of none is equivalent to a size check only and async will not detect a change in timestamp Default sha1 sparse for local storage none f...

Page 242: ...are added to the directory after the start of the async session but not existing files With no scan Aspera Sync relies entirely on file system notifications to detect changes As a result if a directo...

Page 243: ...not synchronized when only the ACL is modified or when only the ACL and filename are modified ACLs are not preserved for directories On Windows the ACLs that are created for files that are transferred...

Page 244: ...target directory that is inside your source directory remote force stat Force the remote Aspera Sync to retrieve file information even if no changes were detected by scanning or file system notificat...

Page 245: ...iously found file that does not have multiple hardlinks it is considered a rename and the remote file is renamed accordingly Usage note This option can be used only on file systems with persistent ino...

Page 246: ...d specially in the argument to r or remote dir W token_string token token_string Use the specified authorization token The token type sync push sync pull or sync bidi must match the direction push pul...

Page 247: ...eads 4 R c logs 200 d c data r bobcat 192 168 4 24 C data K push l 500m Details Specifying the logging locations L and R is optional Adding 200 to the end of the log directory value allows the logs to...

Page 248: ...kipped SYNCHRONIZED del file deleted SYNCHRONIZED ddp dedup duplicate files present SYNCHRONIZED exs file exists SYNCHRONIZED mov file has changed renamed moved or different attributes Include and Exc...

Page 249: ...ows FAT or NTFS file systems and macOS HPFS a file system search for DEBUG returns files Debug and debug In contrast async filter rules use exact comparison To match both Debug and debug in a async fi...

Page 250: ...are equivalent to exclude from G Specifying Rules in aspera conf Rules can be specified in aspera conf and applied to sessions run by a specific user or all users as they are for ascp sessions Rules i...

Page 251: ...f abcefg abc abcde abc z abcdef abc d abc abc def adef cdef abcdef ade abc def zdef def 2def bdef def def abc def zdef def 2def cdef def def xxxxx lower def cdef ydef Adef 2def def Globbing Extensions...

Page 252: ...ized 1 Include files under top level directories Raw and Jpg Exclude all others async include Raw include Jpg exclude exclude 2 Same as Example 1 except also include directories starting with at any l...

Page 253: ...76 or 1MB Continuous transfer Bidirectional transfer Example Command async N asyncTwoWay d fio S r admin 192 168 200 218 d mnt fio S w v00d00 l 100M a fair g 1M G 1M C K BIDI Example Output SYNCHRONIZ...

Page 254: ...count becomes an Aspera transfer user 4 Set database and log directories for async These directories must be located in mnt ephemeral data The mnt ephemeral directory is no cost ephemeral storage that...

Page 255: ...Objects in Object Storage Files that are uploaded to metadata compatible storage S3 Google Cloud and Azure can have custom metadata written with them by using the tags or tags64 option The argument i...

Page 256: ...your cluster and click the Access Keys tab Click New and fill in the required information for a description of the fields see the Aspera Transfer Cluster Manager Admin and Usage Guide AoCts See https...

Page 257: ...tch Service enables fast detection and transfer of new and deleted items For more information on using watches with ascp see Transferring and Deleting Files with the Aspera Watch Service on page 216 T...

Page 258: ...d for the user who runs the service For example if you started a Watch Service under root you should see the root daemon listed when you run the following command opt aspera bin aswatchadmin query dae...

Page 259: ...awatchd is used only for pull requests by that user To configure the Watch Service database as the default run the following command asconfigurator x set_node_data async_watchd redis hostname 31415 do...

Page 260: ..._address port domain This setting applies to both the Watch Service and Watch Folders services redis 127 0 0 1 31415 watchd_max_directories max_directories The maximum number of directories that can b...

Page 261: ...file to determine whether or not to use asperawatchd for the session To pull files start a Aspera Sync session with the K pull option For example async N watch_pull d data D1 r adminuser 10 0 0 1 dat...

Page 262: ...ta R11 K BIDI In this example the client on Host A starts the Aspera Sync session The asperawatchd service on Host B 10 54 44 194 scans the data D1 directory mounted by Host A and passes the snapshot...

Page 263: ...ir path Specify the local Aspera Sync directory E number erase number Delete the specified file record by number F force Allow changes while database is in use f file info Report the status of all fil...

Page 264: ...are older This option is only applied if async has been run using the exclude dirs older than option v verbose Increase the verbosity of summary s or file info f x init Delete all file system snapsho...

Page 265: ...ync but is in error for the underlying ascp process For example when async is run with checksum none and access to the file is denied async does not open the file to calculate a checksum so it does no...

Page 266: ...n You must run your Aspera Sync session to or from a computer with an operating system that supports continuous mode Continuous Aspera Sync Direction Supported Aspera Sync Client OS Supported Aspera S...

Page 267: ...and you want to synchronize the following directory and files on both computers My_documents Document1 Document2 Document3 If Document2 is changed on both computer A and computer B then when you run t...

Page 268: ...u to resolve the original conflict after synchronization Requires access to only one endpoint If you only have access to one endpoint want to preserve changes on both sides but do not want to resolve...

Page 269: ...mand in the ssh folder The program prompts you for the key pair s filename Press ENTER to use the default name id_rsa For a passphrase you can either enter a password or press return twice to leave it...

Page 270: ...ents of the directory media wmv Exclude files within the directory Exclude all other directories Preserve the owner and group ID Preserve access and modification time stamps on files No encryption Tra...

Page 271: ...rom file I include from file Include filter text file with paths for inclusion See Include and Exclude Filtering Rules on page 248 exclude from file E exclude from file Exclude filter text file with p...

Page 272: ...tures and functionality An HTTPS by default port 9092 and HTTP by default port 9091 interface An API that uses JSON data format The API is authenticated and the node daemon uses its own application le...

Page 273: ...e file_restriction restriction Where username is the system user s username is a delimiter and restriction is specific to the storage type and path Storage Type Format Example local storage For Unix l...

Page 274: ...into ssh and rename it authorized_keys or append the public key to authorized_keys if the file already exists cp opt aspera var aspera_tokenauth_id_rsa pub home aspera_user_1 ssh authorized_keys c Ens...

Page 275: ...asperanoded or for Linux systems that use init d service asperanoded restart Node Admin Tool Use the asnodeadmin tool to manage add modify delete and list Node API users Root privileges are required S...

Page 276: ...ope role for bearer create token key length Specify the RSA key length for bearer create user id user_id Specify the user id for bearer create bearer verify Verify bearer token f conf_filename Specify...

Page 277: ...server section of aspera conf Asconfigurator Use the following syntax substituting option with the option from the following table and value with the desired value opt aspera bin asconfigurator x set_...

Page 278: ...rt asperanoded transfers_retry_duration transfers_retry_duration If a transfer fails node will try to restart it for the specified time default 20m If a transfer restarts and makes some progress then...

Page 279: ...415 Before changing this value you should back up your database See Backing up and Restoring the Node User Database Records on page 282 Restart asperanoded and the Redis database ssl_ciphers ssl_ciphe...

Page 280: ...ery transfers that are associated with this access key through the events endpoint The server configuration can be overridden by the access key configuration This option must be enabled for event repo...

Page 281: ...speranoded restart Reload the Node Configuration sudo opt aspera bin asnodeadmin reload Restart asperanoded and the Redis database 1 Stop asperanoded systemctl stop asperanoded or for Linux systems th...

Page 282: ...e Redis database sudo opt aspera bin asnodeadmin r filepath database backup Note If you do not want to keep users that have been added since the last backup operation delete them after performing the...

Page 283: ...de is using the default port for the Redis database port 31415 If your deployment uses a different port for Redis substitute it in the commands accordingly 1 Verify that the original node and new node...

Page 284: ...lude pem crt cer and key and are Base 64 encoded ASCII files containing BEGIN CERTIFICATE and END CERTIFICATE statements Server certificates intermediate certificates and private keys can all be put i...

Page 285: ...signed certificate Note Some certificate authorities provide a CSR generation tool on their website For additional information check with your CA 4 If required generate a self signed certificate You m...

Page 286: ...icate or certificate bundle root certificate with chained or intermediary certificates from an authorized Certificate Authority For instructions on generating an SSL certificate see Setting up SSL for...

Page 287: ...file pem cert_file cert file for asperanoded server Installing the SSL Certificates 1 Back up the default private key and self signed certificate using the following commands cd opt aspera etc cp aspe...

Page 288: ...t examples Success The following sample output shows that verification was successful because verify return is 0 depth 2 C US O VeriSign Inc OU VeriSign Trust Network OU c 2006 VeriSign Inc For author...

Page 289: ...thentication and Authorization Introduction to Aspera Authentication and Authorization HST Server can be configured to support SSH or HTTPS authentication and authorization for browsing and transfers...

Page 290: ...ers or groups are configured to require token authorization only transfers initiated with a valid token transfer token basic token or bearer token are allowed to transfer to or from the server Token a...

Page 291: ...ng purposes For more information on astokengen see Transfer Token Generation astokengen on page 293 Prerequisites In order to create transfer tokens with the Node API you must set up HST Server for th...

Page 292: ...ths destination_root http serengeti com 9091 files upload_setup The response output is the following from which you extract the token string ATV7_HtfhDa JwWfc6RkTwhkDUqjHeLQePiOHjIS254_LJ14_7VTA HTTP...

Page 293: ...users to generate and decode transfer tokens Unless you are creating a transfer token for an Ascp 4 session which requires that you use astokengen with the full paths option Aspera recommends using th...

Page 294: ...download token Each pair of lines encodes one source and one destination and blank lines are ignored For example monday first_thing txt archive monday texts first_thing monday next_thing txt archive...

Page 295: ...ause astokengen to fail Paired upload The destination is prepended to the destinations in the paired list file and they are encoded into the token The destinations are in the odd numbered lines of the...

Page 296: ...triction configured in aspera conf rather than a docroot If a docroot is configured access key creation and use fails Access keys must specify the storage path Although they can be created with no sto...

Page 297: ...eys d access_key_config json where access_key_config json is the access key configuration file For example curl ki u nodeadmin superP 55wOrD X POST https localhost 9092 access_keys d nodeadmin ak_clie...

Page 298: ...transfer Optional JSON object The transfer configuration object Available as of 3 8 0 cipher Optional String The encryption mode and minimum cipher key length allowed by the server for transfers that...

Page 299: ...g network or storage capacity if the client also requests a high minimum transfer rate that is not capped by the server This can decrease transfer performance and cause problems on the target storage...

Page 300: ...aej_logging Optional Boolean Set to true to enable reporting to the IBM Aspera on Cloud Activity app The access key configuration overrides the server configuration This option must be enabled for ac...

Page 301: ...id external_id assume_role_session_name session_name Where path includes the bucket and file path If server side encryption is set to AWS_KMS then server_side_encryption_aws_kms_key_id is required and...

Page 302: ...uch as https company blob core windows net temp sv 2014 02 14 sr c sig yfew 79uXE 3D st 2015 07 29T07 3A00 3A00Z se 2018 08 06T07 3A00 3A00Z sp rwdl Azure Files storage type azure files path share pat...

Page 303: ...40developer gserviceaccount com IBM Cloud Object Storage COS S3 storage type ibm s3 path bucket path endpoint s3 api us geo objectstorage service networklayer com credentials type key access_key_id k...

Page 304: ...ess to a specific area of a storage and authenticates that user to the storage Basic tokens are less restrictive than transfer tokens They can be used to transfer with any Aspera server that supports...

Page 305: ...l other Aspera servers too To create a bearer token with asnodeadmin run the following command as a user with admin root permissions If you do not specify an SSL key file or directory you are asked if...

Page 306: ...n your HST Server installation Overview Deploying HST Server as a high availability cluster enables you to leverage the high speed transfer capabilities of Aspera with continuous availability and auto...

Page 307: ...hysical virtual or container on which IBM Aspera High Speed Transfer Server is installed Glibc 2 5 or higher SSH Server If you are using OpenSSH version 5 2 or higher is recommended These instructions...

Page 308: ...s create or edit an existing opt aspera etc redis conf with the following values slave of primary_ip_address 31415 bind private_ip_address port 31415 daemonize yes pidfile opt aspera var run redis 314...

Page 309: ...era etc redis sentinel log Save your changes and close the file 5 On each node configure HAProxy a Open opt aspera etc haproxy haproxy cfg template http haproxy 1wt eu download 1 4 doc configuration t...

Page 310: ...y the configuration opt aspera sbin haproxy f opt aspera etc haproxy haproxy cfg c e Configure iptables to ACCEPT the Redis IP addresses Your cluster is now configured You can now launch and test it S...

Page 311: ...lly can be cumbersome and error prone because correct syntax and structure are strictly enforced The asconfigurator utility enables you to edit aspera conf through commands and parses validates and wr...

Page 312: ...feature settings for use with the Node API For parameters and values see Server Configurations on page 325 set_http_server_data Sets data in the HTTP fallback server section For parameters and values...

Page 313: ...x command parameter value fitness fitness_rule fitness_template Fitness Rule Example Description cookie cookie wilcard_template The parameter value is applied if the cookie passed from the application...

Page 314: ...bling a Vlink with an ID of 101 and a capacity of 100Mb s asconfigurator x set_trunk_data id 101 trunk_on true trunk_capacity 100000 Allowing only encrypted transfers asconfigurator x set_node_data tr...

Page 315: ...ase section c Outputs configurations set in the central server section t Outputs configurations set in the HTTP server section a Outputs configurations set in all sections except the user and group se...

Page 316: ...incoming transfers Values String authorization_transfer_out_external_provider_url The URL of the external authorization provider for outgoing transfers Values String authorization_transfer_in_external...

Page 317: ...sfer_out_bandwidth_flow_target_rate_lock A value of false allows users to adjust the transfer rate for outgoing transfers A value of true prevents users from adjusting the transfer rate for outgoing t...

Page 318: ...s users to adjust the bandwidth policy for outgoing transfers A value of true prevents users from adjusting the bandwidth policy for outgoing transfers Values false default true transfer_in_bandwidth_...

Page 319: ...he location path where file manifests are created Values Absolute path pre_calculate_job_size The policy of calculating total job size before a transfer If set to any the client configuration is follo...

Page 320: ...by Aspera users Values Absolute path read_allowed Whether users are allowed to transfer files from the docroot in other words download from the docroot Values true default false write_allowed Whether...

Page 321: ...mmand opt aspera bin asuserdata Vlink Configurations trunk_id The ID of the Vlink Values Number 1 255 trunk_on Whether the Vlink is enabled true or disabled false Values true false trunk_capacity The...

Page 322: ...error Values ignore default exit compact_on_startup Whether to compact the local transfer history database on startup note that this may take awhile Values ignore default exit files_per_session The n...

Page 323: ...lows the HTTP Fallback server to accept transfer requests on all network interfaces Values Network interface address default 0 0 0 0 restartable_transfers Whether interrupted transfers should resume a...

Page 324: ...password The password for the database server Values String database_name The name of the database used to store Aspera transfer data Values String threads The number of parallel connections used for...

Page 325: ...t 1 ignore_empty_files Whether to block the logging of zero byte files true or not false Values false default true ignore_skipped_files Whether to block the logging of skipped files true or not false...

Page 326: ...er HTTP is enabled for asperanoded on the port configured for http_port true or not false Values false default true enable_https Whether HTTPS is enabled for asperanoded on the port configured for htt...

Page 327: ...documentation for the default list of ciphers Values Colon delimited list ssl_protocol The minimum allowed SSL protocol Higher security protocols are always allowed tlsv1 default tlsv1 1 tlsv1 2 Aspe...

Page 328: ...e first character is a separator preferably a which can be used to set multiple hosts For example 10 0 23 123 33001 10 0 23 124 33001 10 0 23 125 33001 Values Character separator IP address Character...

Page 329: ...e following command opt aspera bin asuserdata Parameters and Values transport_cipher The encryption cipher to use for transfers Values aes 128 default aes 192 aes 256 none ssl_ciphers The list of SSL...

Page 330: ...gnore the permission denied message after entering the password which is discussed in next steps 4 Applied authentication method is enabled in SSH If you can establish a SSH connection but it returns...

Page 331: ...he first time running htpasswd to create the webpasswd file Do not use the c option otherwise If you still encounter connection problems after going through these steps contact Technical Support on pa...

Page 332: ...e default of 10 MB For information on other logging configuration options see Server Logging Configuration for Ascp and Ascp 4 on page 78 Logging settings are configured by running asconfigurator comm...

Page 333: ...ped or if you have modified the central_server or database sections in aspera conf then you need to restart the service Run the following command in a Terminal window to restart asperacentral systemct...

Page 334: ...e that can include a substitutional string Supported strings name home The pathname can be in URI format special characters must be URL encoded A set of file system filters that use as a wildcard and...

Page 335: ...curing the Aspera Application Securing Content in your Workflow Securing the Systems that Run Aspera Software The systems that run Aspera software can be secured by keeping them up to date by applying...

Page 336: ...web UI you must also update the SshPort value in the WEB section of aspera conf For details see Configuring your Web UI Settings on page 25 Once this setting takes effect Aspera clients must set the...

Page 337: ...mptyPasswords no e Disable root login CAUTION This step disables root access Make sure that you have at least one user account with sudo privileges before continuing otherwise you may not have access...

Page 338: ...eract with the servers The instructions for Shares 1 9 x and Shares 2 x are slightly different see the section for your version HST Server 1 Restrict user permissions with aspshell By default all syst...

Page 339: ...Files azure files Google Cloud Storage gs Hadoop HDFS hdfs The is a delimiter and you can add additional restrictions For example to restrict the system user xfer to s3 s3 amazonaws com bucket_xyz fol...

Page 340: ...ryption If you require higher encryption change this value by running the following command asconfigurator x set_client_data transport_cipher value You can also specify the encryption level in the com...

Page 341: ...self signed certificates Aspera recommends installing valid signed certificates These are required for some applications Securing Content in your Workflow 1 If your workflow allows enable server side...

Page 342: ...tor x set_group_data group_name group_name transfer_encryption_content_protection_secret passphrase Important If the EAR password is lost or aspera conf is compromised you cannot access the data on th...

Page 343: ...an be unencrypted To encrypt a file before moving it to a computer with network access run the following commands to set the encryption password and encrypt the file export ASPERA_SCP_FILEPASS passwor...

Page 344: ...sequential or random Default is sequential optional When set to sequential file size is calculated as size N 1 increment Where N is the file index for the first file N is one When set to random file...

Page 345: ...disk The I O throughput the disk bus architecture such as RAID IDE SCSI ATA and Fiber Channel Network I O The interface card the internal bus of the computer CPU Overall CPU performance affects the t...

Page 346: ...an leave some blank For some fields there will be a default value If you enter the field will be left blank Country Name 2 letter code US Your_2_letter_ISO_country_code State or Province Name full nam...

Page 347: ...key in any directory as long as the paths are updated in your configuration file For additional information see Enable SSL Apache Enable SSL Apache Install and enable an SSL certificate for your HST S...

Page 348: ...ebian 7 or older Ubuntu 14 10 or older sudo service apache2 restart CentOS 6 RHEL 6 sudo service httpd restart 4 Test your SSL connection Go to https your server ip or name to test your SSL setup This...

Page 349: ...e application window The five logging levels to select from are Off Error Warn Info and Debug The system default is Info Redirecting Aspera Logging to a Different Location On Linux systems the applica...

Page 350: ...n the case of Red Hat or CentOS 6 X service rsyslog restart Your Aspera log messages now appear in var log aspera log instead of var log messages SLES Suse systems On SLES Suse systems the transfer lo...

Page 351: ...slog as follows var log messages var log secure var log maillog var log spooler var log boot log var log cron var log aspera log sharedscripts postrotate bin kill HUP cat var run syslogd pid 2 dev nul...

Page 352: ...utable yes Ascp and Aspera Sync update the destination file from mutable to immutable However if the source file is changed back to mutable immutable no the change cannot be applied to the destination...

Page 353: ...ny All rights reserved Licensed Materials Property of IBM 5725 S58 Copyright IBM Corp 2007 2019 Used under license US Government Users Restricted Rights Use duplication or disclosure restricted by GSA...

Search results

Summary of Contents for Aspera HST

Reviews:

Related manuals for Aspera HST

Brands by name

Popular brands

IBM Aspera HST, Admin Manual

Search results

Summary of Contents for Aspera HST

Reviews:

Related manuals for Aspera HST

Brands by name

Popular brands