IBM Aspera HST Скачать руководство пользователя страница 306

Страница: 306 / 353

| Highly-Available Redis Backend for an HST Server Cluster |

306

Option

Required

Type

Description

--expires-at

Optional

UTC time

The expiration date of

the bearer token in UTC

format. For example,

2016-06-23T13:21:58.453Z.

Default expiration is 1 hour

after token creation time.

--scope-role

Optional

String

The access level of the

bearer token. Value can be

admin

(default) or

user

admin

can change the

access key configuration,

user

cannot.

--token-key-length

Optional

Double

The length of the RSA

key. Must be a power of

2 between 1024 bits (128

bytes) and 16384 bits

(2048 bytes). Default key

length is 4096 bits.

Highly-Available Redis Backend for an HST Server Cluster

A guide to configuring a highly-available Redis database that is used by all HST Servers in a cluster.

Introduction

HST Server can be deployed as a high-availability cluster. This is done by configuring a Redis database cluster and

using HAProxy to connect the HST Servers to the Redis database. The Redis and HAProxy applications are included

in your HST Server installation.

Overview

Deploying HST Server as a high-availability cluster enables you to leverage the high-speed transfer capabilities of

Aspera with continuous availability and automatic failover if one node goes down.
A cluster consists of

nodes

. A node is a machine (physical, virtual, or container) on which IBM Aspera High-Speed

Transfer Server and Redis are installed. A Redis node can be a primary (also called a "master") or replica ("slave").

Cluster Architecture

In an HST Server cluster with a highly-available Redis backend:
• Each node has HST Server, Redis, and HAProxy. HST Server uses the

Redis database

to manage security and

transfers. In a cluster configuration, all HST Servers use the Redis database on the primary node, which is cloned

to the replica nodes.

• HST Server uses

Hiredis

, a Redis client, to interact with the Redis database.

• To direct Hiredis requests from all the HST Servers to the primary Redis node, the cluster uses

HAProxy

, a TCP

proxy, to identify which Redis nodes are running and which is the master.

•

Redis Sentinel

monitors the status of the Redis databases, communicates between Redis nodes, and automatically

promotes a replica to primary if the primary is failing.

Содержание Aspera HST

Страница 1: ...High Speed Transfer Server Admin Guide 3 9 1 PowerLinux Revision 1978 Generated 04 05 2019 10 17...

Страница 2: ...iated Transfer 20 Updating the Product License 21 Uninstalling 21 Set up the HST Server Web UI 22 Configuring the Apache Server to Host the HST Server Web UI 22 Configuring your Web UI Settings 25 Cus...

Страница 3: ...les 97 ascp Transferring from the Command Line with Ascp 99 Ascp Command Reference 99 Ascp General Examples 114 Ascp File Manipulation Examples 116 Ascp Transfers with Object Storage and HDFS 118 Tran...

Страница 4: ...ice Configuration 213 Setting Custom Watch Scan Periods 215 Managing Watch Subscriptions 215 Transferring and Deleting Files with the Aspera Watch Service 216 Aspera Sync 218 Introduction 218 Overview...

Страница 5: ...our Nodes 284 Installing SSL Certificates 286 Authentication and Authorization 289 Introduction to Aspera Authentication and Authorization 289 Require Token Authorization Set from the Command Line 290...

Страница 6: ...ervices 333 Docroot vs File Restriction 334 Aspera Ecosystem Security Best Practices 335 Securing the Systems that Run Aspera Software 335 Securing the Aspera Applications 338 Securing Content in your...

Страница 7: ...cipates in transfers The server can be an on premises installation of HST Server IBM Aspera High Speed Transfer Endpoint which permits one client connection a HST Server installed as part of IBM Asper...

Страница 8: ...that storage HST Server can be incorporated into a scalable Aspera data transfer ecosystem that meets your needs Your Aspera server can be monitored and managed by IBM Aspera Console and added as a no...

Страница 9: ...r destination for authorized transfers Your server can also take the role of a client and connect to other Aspera servers to initiate transfers The following steps describe how to prepare your system...

Страница 10: ...olders to clients when they are added to a specific folder on the server see Introduction to Watch Folders and the Aspera Watch Service on page 159 If you want to enable server based clients to synchr...

Страница 11: ...lders Watch Folders Aspera Sync Supported platforms Windows only Windows macOS Linux AIX Solaris Linux on z Systems BSD Isilon Windows macOS Linux AIX Solaris Linux on z Systems BSD Additional license...

Страница 12: ...e system change in the Hot Folder is detected On a user specified schedule Immediate as soon as a difference between snapshots is detected Immediate in continuous mode or when using Aspera Sync with a...

Страница 13: ...hat supports LE SSH Server Version 7 0 or higher is recommended To use the Node API The line 127 0 0 1 localhost must appear in the hosts file etc hosts For UNIX based nodes SELinux must be set to per...

Страница 14: ...d dependencies are installed with your Aspera application by installing the product with a yum install yum nogpgcheck install path_to_installer aspera hsts version rpm On some CentOS 7 and Fedora syst...

Страница 15: ...tication to yes To allow password authentication set PasswordAuthentication to yes For example PubkeyAuthentication yes PasswordAuthentication yes c Save the file then reload the SSH service d Restart...

Страница 16: ...ee Controlling Bandwidth Usage with Virtual Links Command Line on page 58 Remote Client Machines Typically consumer and business firewalls allow direct outbound connections from client computers on TC...

Страница 17: ...ection tab click Show Advanced Settings and enter the SSH port number in the SSH Port TCP field Command line Clients running FASP transfers from the command line can specify the port by using the P 33...

Страница 18: ...ure authentication methods add or uncomment PubkeyAuthentication yes and comment out PasswordAuthentication yes PubkeyAuthentication yes PasswordAuthentication yes PasswordAuthentication no Note If yo...

Страница 19: ...ku sshd 1496 Failed password for invalid user alex from 1 2 3 4 port 1585 ssh2 Mar 14 23 25 52 sku sshd 1496 Failed password for invalid user alice from 1 2 3 4 port 1585 ssh2 If you identify attacks...

Страница 20: ...h_host_key_path 2 Restart the node service to activate your changes Run the following commands to restart asperanoded systemctl restart asperanoded or for Linux systems that use init d service asperan...

Страница 21: ...cp T tmp 100MB aspera demo asperasoft com Upload Updating the Product License Update your product license from the command line 1 Open the license file with write permission opt aspera etc aspera lice...

Страница 22: ...tions describe how to configure your system s Apache server to host HST Server s web UI The Apache files might be located in different paths or your Apache server could require additional settings dep...

Страница 23: ...on to SSH authentication HST Server uses Apache s authentication to authorize web UI access To set up a system user asp1 in this example for Apache authentication run the htpasswd command below Note O...

Страница 24: ...Paste the output generated when you ran the enablesecure script as described above BEGIN IBM Aspera High Speed Transfer Server The user account that runs the web server will impersonate the logged in...

Страница 25: ...nging it to TCP 33001 as described in Securing your SSH Server The default configuration example above assumes your SSH port is set to TCP 33001 The table below provides descriptions of all web UI con...

Страница 26: ...If the minimum version is not installed a message is displayed that indicates the minimum version required and provides a download link This option takes the value in the format of the Connect version...

Страница 27: ...wing locations Header opt aspera var webtools aspdir header html Footer opt aspera var webtools aspdir footer html 2 Modify the header and footer then save your changes Testing the Web UI Once your HS...

Страница 28: ...n file_system access paths path absolute sandbox name absolute path paths access file_system default CONF To add the settings to aspera conf manually open it from the following directory opt aspera et...

Страница 29: ...back HTTP fallback serves as a secondary transfer method when the Internet connectivity required for Aspera FASP transfers UDP port 33001 by default is unavailable When HTTP fallback is enabled and UD...

Страница 30: ...Save and close the file d Confirm that aspera conf is formed correctly Validate the aspera conf file using the asuserdata utility opt aspera bin asuserdata v 2 Configure HTTP HTTPS fallback settings R...

Страница 31: ...o allow interrupted transfers to resume from the point of interruption true or false true Session Activity Timeout Any value greater than 0 sets the amount of time in seconds that the HTTP fallback se...

Страница 32: ...20 recommended This adds or updates the encryption_key value in the authorization section Important After changing your Aspera token settings either in aspera conf or the GUI you must restart asperaht...

Страница 33: ...nfigured as Aspera transfer users before clients can browse the server file system or run FASP transfers to and from the server When creating transfer users you can also specify user specific settings...

Страница 34: ...user s home directory home Documents 3 For server security Aspera recommends restricting users read write and browse permissions Users are given read write and browse permissions to their docroot by...

Страница 35: ...rname absolute docroot asconfigurator x set_user_data user_name username transfer_in_bandwidth_flow_target_rate_default rate asconfigurator x set_user_data user_name username transfer_out_bandwidth_fl...

Страница 36: ...is a member of multiple groups the precedence setting can be used to determine priority aspera conf Authorization Configuration on page 40 Connection permissions token key and encryption requirements...

Страница 37: ...ealm users user specific settings users groups group Each group tag contains a group s profile name aspgroup name The group name precedence 0 precedence Group precedence authorization authorization Au...

Страница 38: ...the Server Public key authentication is an alternative to password authentication providing a more secure authentication method that allows users to avoid entering or storing a password or sending it...

Страница 39: ...must have at least one Aspera transfer user a system user account that is configured to authenticate Aspera transfers configured on it If any of the following connection tests fail see Clients Can t...

Страница 40: ...t Address ip_address 10 0 0 2 Destination Folder Set the destination path relative to the transfer user s docroot destination dir In this example the files are transferred to the dir folder in the doc...

Страница 41: ...yption_key Token Encryption Key filename_hash filename_hash Token Filename Hash life_seconds 86400 life_seconds Token Life seconds token authorization 3 Edit settings as needed Authorization Settings...

Страница 42: ...t text string blank Token Encryption Cipher Set the cipher used to generate encrypted transfer tokens aes 128 aes 192 or aes 256 aes 128 Token Encryption Key Set the secret text phrase that is used to...

Страница 43: ...Priority Default lock false lock Incoming Priority Lock priority network_rc module delay module Incoming Rate Control Module tcp_friendly false tcp_friendly Incoming TCP Friendly Mode predictor unset...

Страница 44: ...tions validation_file_start none validation_file_start Validation File Start validation_file_stop none validation_file_stop Validation File Stop validation_session_start none validation_session_start...

Страница 45: ...rver If the client requested minimum rate exceeds network or storage capacity this can decrease transfer performance and cause problems on the target storage positive integer or unlimited unlimited In...

Страница 46: ...w bandwidth policy are allowed All others are rejected Incoming Bandwidth Policy Default The default bandwidth policy for incoming transfers Clients can override the default policy if they specify a p...

Страница 47: ...l priority setting Use the value 0 to unset this option 1 to allow high priority 2 to enforce normal priority 0 1 or 2 2 Incoming Priority Lock To disallow your clients change the priority set the val...

Страница 48: ...ng network congestion When set to unset the client specified predictor is used and if the client does not specify a predictor then none is used For more information see Increasing Transfer Performance...

Страница 49: ...ting the minimum rate cap to zero Transfers do not slow below the client s requested minimum rate unless the minimum rate is capped on the server If the client requested minimum rate exceeds network o...

Страница 50: ...width policies are allowed Transfers that request fixed bandwidth policy are rejected low Only transfers that use a low bandwidth policy are allowed All others are rejected high fair low or any any Ou...

Страница 51: ...o unset this option 1 to allow high priority 2 to enforce normal priority 0 1 or 2 1 Outgoing Priority Default The initial priority setting Use the value 0 to unset this option 1 to allow high priorit...

Страница 52: ...rease transfer rate stability and throughput by predicting network congestion When set to unset the client specified predictor is used and if the client does not specify a predictor then none is used...

Страница 53: ...and supports two encryption modes cipher feedback mode CFB and Galois counter mode GCM The GCM mode encrypts data faster and increases transfer speeds compared to the CFB mode but the server must supp...

Страница 54: ...d is set to true in aspera conf and you use passphrase protected SSH keys you must use keys generated by running ssh keygen in a FIPS enabled system or convert existing keys to a FIPS compatible forma...

Страница 55: ...ession below the input value The default of 0 will cause the Aspera sender to use its default internal buffer size which may be different for different operating systems positive integer 0 Minimum Soc...

Страница 56: ...For threshold validation the file transfer might complete before the file threshold validation response comes back because ascp doesn t pause file transfers during file threshold validation therefore...

Страница 57: ...o Lua Action Script must be defined if any of the following values are set to lua_script Run at File Start Run at File Stop Run at Session Start Run at Session Stop Run when Crossing File Threshold If...

Страница 58: ...users create a Vlink with a 10 Mbps capacity and assign it to outgoing transfers for those three users If the three users are running download sessions that already use 10 Mbps and another download is...

Страница 59: ...era conf CONF version 2 trunks trunk id 108 id Vlink ID name 50Mbps cap name Vlink Name capacity schedule format ranges 50000 schedule Capacity capacity on true on On trunk trunks CONF The capacity of...

Страница 60: ...2 default transfer out bandwidth aggregate trunk_id 108 trunk_id Vlink 108 for the default outgoing sessions aggregate bandwidth out in in transfer default aaa realms realm users user name aspera_user...

Страница 61: ...x set_trunk_data id 108 trunk_capacity 88000 trunk_on true asconfigurator x set_trunk_data id 109 trunk_capacity 99000 trunk_on true asconfigurator x set_node_data transfer_in_bandwidth_aggregate_trun...

Страница 62: ...fference method bezier A quadratic Bezier extrapolation ets An error trend seasonality model Based on internal testing fd31 is considered the most effective and robust but other RTT predictors might p...

Страница 63: ...a docroot For more information see Docroot vs File Restriction on page 334 Configuration methods These instructions describe how to manually modify aspera conf You can also add and edit these paramet...

Страница 64: ...file_suffix partial partial_file_suffix Partial File Suffix file_checksum any file_checksum File Checksum Method file_system 3 Edit settings as needed File System Settings Reference Field Description...

Страница 65: ...ictions do not start with the user can access any file that matches any one of the no restrictions Format examples For a specific folder For the drive root For ICOS S3 storage s3 my_vault To exclude a...

Страница 66: ...teger 0 Number of Dir Scanning Threads Set the number of threads the Aspera sender uses to scan directory contents It takes effect on both client and server when acting as a sender The default of zero...

Страница 67: ...ed for resuming incomplete transfers Each data file in progress will have a corresponding metadata file with the same name plus the resume suffix specified by the receiver Metadata files in the source...

Страница 68: ...calculating job size before transferring Set to no to disable calculating job size before transferring Set to any to follow client configurations yes no or any any Convert Restricted Windows Characte...

Страница 69: ...to manage the asperacentral database Configuration methods These instructions describe how to manually modify aspera conf You can also add and edit these parameters using asconfigurator commands For m...

Страница 70: ...torage Path Valid system path If the application is installed in the default location then the path is the following Maximum Age seconds Maximum allowable age in seconds of data to be retained in the...

Страница 71: ...listed in aspera conf Filtering is a process of exclusion and include rules override exclude rules that follow them Include rules cannot add back files that are excluded by a preceding exclude rule I...

Страница 72: ...attern abc f matches abcdef but not abcdefg For details on using wildcards and special characters to build rule patterns see Using Filters to Include and Exclude Files on page 126 Set Rules Filter rul...

Страница 73: ...or user asp1 asconfigurator x set_user_data user_name asp1 file_filters abc wxy tuv abc def Results in aspera conf aaa realms realm users user name asp1 name file_system filters filter abc wxy tuv fil...

Страница 74: ...ansfers FASP and HTTP fallback transfers Requirements If the following requirements are not met then the server can have both encrypted and unencrypted content This can cause file corruption on the se...

Страница 75: ...d Connect Server or HST Endpoint formerly Point to Point Client version 3 4 2 or higher The transfer must be encrypted Encryption is enabled by default The user on the destination can calculate a chec...

Страница 76: ...t source path at the sender Note File manifests can be stored only locally Thus if you are using S3 or other non local storage you must specify a local manifest path Enabling checksum reporting by edi...

Страница 77: ...ieve the checksum that was calculated by Aspera as the file was transferred If you specified a file manifest and file manifest path as part of an ascp transfer or pre post processing script the checks...

Страница 78: ...port user specific logging settings If the client specifies a log directory on the server using R remote_log_dir or the location and size of the local log directory using L local_log_dir size then the...

Страница 79: ...The full path to the logging directory Applies only to ascp transfers log_size The size of the log file in MB at which it is rotated the oldest information is overwritten by the newest information Def...

Страница 80: ...transfer file validation is run as soon as the client uploads a to HST Server The transfer is reported as complete and then the validation is run The validation script uses the Aspera Reliable Query A...

Страница 81: ...max_result sets a batch size for how many files are collected for validation by each POST request and cannot exceed 1000 The POST request retrieves the files that are to_be_validated updates their st...

Страница 82: ...session_uuid session_uuid file_id file_id status error error_code error_number error_description error_string https server_name 9092 services rest transfers v1 files For example the body of a PUT req...

Страница 83: ...ntly be executed by an external product that integrates with an Aspera product Inline file validation is a feature that enables file content to be validated while the file is in transit as well as whe...

Страница 84: ...s IP address and port and the servlet name URL handler found in web xml This adds the path to the transfer section of aspera conf For example transfer validation_uri http 127 0 0 1 8080 SimpleValidat...

Страница 85: ...2 defining values in aspera conf For more information on the output of your inline validation see Inline File Validation with URI on page 85 or Inline File Validation with Lua Script on page 87 Inline...

Страница 86: ...t javax servlet http HttpServlet import javax servlet http HttpServletRequest import javax servlet http HttpServletResponse import java io BufferedReader import java io IOException WebServlet name Sim...

Страница 87: ...cript defined in aspera conf The parameters for Lua calls are passed to Lua scripts by using the array env_table The following is an example request body env_table startstop running env_table xfer_id...

Страница 88: ...tring AES128 ANY or NONE cookie The cookie sent to the client system String manifest_file Path to manifest file which contains a list of transferred files The command for this in ascp is file manifest...

Страница 89: ...S_IFIFO S_IFSOCK S_IFLNK Block stream Custom Unknown stat_data mode format Windows format Linux format stat_data mode filemode format based on mode format above stat_data uid uid stat_data gid gid st...

Страница 90: ...ransfer events Session start Session end Start of each individual file transfer in the session End of each individual file transfer in the session The aspera prepost script can also execute additional...

Страница 91: ...an be written directly into the script file aspera prepost For example to add the custom script script1 pl to your pre post script insert the following line into aspera prepost perl script1 pl Pre Pos...

Страница 92: ...escription Values Example FILE1 The first file string FILE1 first file FILE2 The second file string FILE2 second file FILECOUNT The number of files positive integer FILECOUNT 5 FILELAST The last file...

Страница 93: ...itive integer STARTBYTE 100000 Pre Post Script Examples The following pre processing and post processing script examples demonstrate how Aspera prepost environment variables are used to achieve differ...

Страница 94: ...y must be cached bin bash TARGET aspera 10 10 10 10 tmp RATE 10m export ASPERA_SCP_PASS aspera if TYPE File then if STARTSTOP Stop then if STATE success then if DIRECTION recv then logger plocal2 info...

Страница 95: ...repare the email notification configuration template Open the aspera conf file opt aspera etc aspera conf Locate or create the section EMAILNOTIF EMAILNOTIF CONF version 2 EMAILNOTIF MAILLISTS mylist...

Страница 96: ...in FILTER FILTER defines email notification conditional filters When the conditions are met a customized email is sent to the indicated mailing list Multiple filters are allowed The values in the filt...

Страница 97: ...SENDONSESSION yes yes no SENDONSTOP yes SENDONFILE Send email for each file within a session yes no SENDONFILE yes Email Notification Examples Use the following examples to craft your own email notif...

Страница 98: ...e is sent to mediaGroup When a regular transfer occurs files are sent to upload a different notification is sent to mediaLead and adminGroup EMAILNOTIF MAILLISTS mediaGroup johndoe companyemail com ja...

Страница 99: ...e src_host source1 source2 username dest_host dest_path username The username of the Aspera transfer user can be specified as part of the source or destination whichever is the remote server It can al...

Страница 100: ...place each backslash in the UNC path with a forward slash For example if the UNC path is 192 168 0 10 temp change it to 192 168 0 10 temp This format can be used with any client side operating system...

Страница 101: ...01 0 4137 9e50 201b 63d3 ba92 da path or host fe80 21b 21ff fe1c 5072 eth1 range_start range_end Transfer only part of a file range_start is the first byte to send and range_end is the last If either...

Страница 102: ...does not support GCM mode in this case you cannot request GCM mode encryption When the server setting is none you must use none Transfer requests that specify an encryption cipher are refused by the s...

Страница 103: ...e transfer delete any files that exist at the destination but not also at the source The source and destination arguments must be directories that have matching names Do not use with multiple sources...

Страница 104: ...t files when receiving for client side encryption at rest EAR Encrypted files have the file extension aspera env This option requires the encryption decryption passphrase to be set with the environmen...

Страница 105: ...ch source is specified on a separate line with its destination on the line following it Specify destinations relative to the transfer user s docroot Even if a destination is specified as an absolute p...

Страница 106: ...ich might vary by operating system The sending server never uses the read_block_size set in the client s aspera conf h help Display the help text host hostname Transfer to the specified host name or a...

Страница 107: ...d the source files remain in their original location To preserve portions of the file path above the transferred file or directory use this option with src base For an example see Ascp File Manipulati...

Страница 108: ...Rules found in aspera conf are applied before any E and N rules specified on the command line O fasp_port Use the specified UDP port for FASP transfers Default 33001 overwrite never always diff diff...

Страница 109: ...rate to fully utilize the available bandwidth up to the maximum rate When congestion occurs bandwidth is shared fairly by transferring at an even rate The fair policy requires maximum target and mini...

Страница 110: ...Preserve the group information gid or owner information uid of the transferred files These options require the transfer user to be authenticated as a superuser preserve modification time Set the modi...

Страница 111: ...der for the target side of a pull Ascp with mode recv to apply the ACLs remote preserve xattrs native metafile none Like preserve xattrs but used when attributes are stored in a different format on th...

Страница 112: ...urce path includes an embedded passphrase the prefix must also include the embedded passphrase otherwise it will not match For examples see Ascp File Manipulation Examples on page 116 symbolic links f...

Страница 113: ...ng a version of ascp that is older than 3 3 in which case the client setting is used If the pre 3 3 client does not set Z the datagram size is the discovered MTU and the server logs the message LOG Pe...

Страница 114: ...00m O 42000 local dir files user 10 0 0 2 remote dir Public key authentication Transfer with public key authentication using the key file home dir ssh aspera_user_1 key local dir files ascp l 10m i ss...

Страница 115: ...ra env from the server 10 0 0 2 and decrypt while transferring export ASPERA_SCP_FILEPASS secRet ascp l 10m file crypt decrypt root 10 0 0 2 remote dir file aspera env local dir Decrypt a downloaded e...

Страница 116: ...tents to a new directory by using the d option Upload the data directory to the server and if it doesn t already exist create the new folder storage2 to contain it resulting in storage2 data at the de...

Страница 117: ...ot copy srcdir to the archive directory Archive on the server ascp move after transfer Archive Pat 10 0 0 1 srcdir C Users Pat Move the source file on the client after it is uploaded to the server and...

Страница 118: ...l Examples on page 114 You are prompted for the transfer user s password when you run an ascp command unless you set the ASPERA_SCP_PASS environment variable or use SSH key authorization With No Docro...

Страница 119: ...e Aspera recommends running ascp transfers with Azure Data Lake Storage with a docroot configured Upload syntax ascp options mode send user username host server_address source_files azu storage_accoun...

Страница 120: ...ver_address s3 access_id secret_key accessor_endpoint vault_na source_files destination_path Download example ascp mode send user bear host s3 asperasoft com s3 3ITI3OIUFEH233 KrcEW AIuwQ 38 123 76 24...

Страница 121: ...O port_1 multi session threashold threshold tags aspera xfer_id transfer_id source_path hostname destination_path ascp C nid_2 ncount l max_rate O port_2 multi session threashold threshold tags aspera...

Страница 122: ...n the following asconfigurator x set_node_data transfer_multi_session_threshold_default threshold Multi Session Transfer Example The following example shows a multi session transfer on a dual core sys...

Страница 123: ...ite diff and overwrite diff older is undefined Single file Transfer Examples Upload 1025 bytes of data from the client stdin to remote dir on the server at 10 0 0 2 Save the data as the file newfile T...

Страница 124: ...rectories are not allowed Only overwrite always or overwrite never are supported with stdio tar The behavior of overwrite diff and overwrite diff older is undefined Offsets are only supported if the d...

Страница 125: ...0m mode recv keepalive M 12345 user username host 10 0 0 2 stdio tar Send the following in through management port 12345 FASPMGR 2 Type START Source tmp myfile1 Destination mynewfile1 FASPMGR 2 Type S...

Страница 126: ...ing rules are configured in aspera conf they are applied before the rules on the command line Filtering is a process of exclusion and N rules override E rules that follow them N cannot add back files...

Страница 127: ...e files are evaluated Example Consider the following command ascp N file2 E file 0 9 images icons user1 examplehost tmp Where images icons is the source If images icons contains file1 file2 and fileA...

Страница 128: ...ern Matches directories only With N no files under matched directories or their subdirectories are included in the transfer All subdirectories are still included although their files will not be inclu...

Страница 129: ...the target is the Upload directory At the prompt enter the password demoaspera 3 Create a destination directory on your computer for example tmp dest 4 Download your files from the demo server to tmp...

Страница 130: ...abc wxy def AAA abc wxy tuv def AAA abc xyz def wxy AAA wxyfile AAA wxy xyx AAA wxy xyxfile 3 Include directories and files that start with wxy if they fall directly under AAA N wxy E AAA Results AAA...

Страница 131: ...AA wxy xyx AAA wxy xyxfile AAA abc def AAA abc def AAA abc wxy def 6 Exclude directories and files starting with wxy but only those found at a specific location in the tree E AAA abc wxy Results AAA a...

Страница 132: ...c links Copy Client only Copy only the symbolic link If a file with the same name exists at the destination the symbolic link does not replace the file Copy force Client only Copy only the symbolic li...

Страница 133: ...asconfigurator x set_user_data user_name username symbolic_links value For more information see aspera conf File System Configuration on page 63 Client Configuration To specify symbolic link handling...

Страница 134: ...ne use the option i private_key_file For example ascp T l 10M m 1M i ssh id_rsa myfile txt jane 10 0 0 2 space In this example you are connecting to the server 10 0 0 2 directory space with the user a...

Страница 135: ...hen set to text a text file is generated that lists all files in each transfer session file_manifest_path file_manifest_path path The location where manifest files are written The location can be an a...

Страница 136: ...ra and find that it is corrupted you can determine when the corruption occurred by comparing the checksum that is reported by Aspera to the checksums of the files on the destination and on the source...

Страница 137: ...an encryption password and the files are uploaded to the server with a aspera env extension Anyone downloading these aspera env files must have the password to decrypt them and decryption can occur a...

Страница 138: ...t ASPERA_SCP_FILEPASS password opt aspera bin asprotect o file1 aspera env file1 To download client side encrypted files without decrypting them immediately run the transfer without decryption enabled...

Страница 139: ...list filepath file manifest none text file manifest path directory file manifest inprogress suffix suffix file pair list filepath G write_size g read_size h help h help i private_key_file_path i priv...

Страница 140: ...file owner gid preserve file owner gid preserve file owner uid preserve file owner uid preserve modification time preserve source access time preserve xattrs mode proxy proxy_url q q R remote_log_dir...

Страница 141: ...s not exist With ascp4 you must specify d otherwise all the files in the file list are written to a single file i SSH key authentication With ascp the argument for i can be just the file name of the p...

Страница 142: ...te regardless of network or storage capacity This can decrease transfer performance and cause problems on the target storage Aspera discourages using the fixed policy except in specific contexts such...

Страница 143: ...t On Windows the only option is skip Symbolic link handling also depends on the server configuration and the transfer direction For more information see Symbolic Link Handling on page 132 5 What are m...

Страница 144: ...ng For more information see the IBM Aspera Streaming for Video User Guide Required Configuration for Multicast to Multicast Streaming The transfer user who authenticates the data multicast stream tran...

Страница 145: ...e specified If a destination path is a URI no docroot upload or local docroot download can be specified The special schemes stdio and stdio tar are supported only on the client side They cannot be use...

Страница 146: ...enticate to a URI destination Ascp 4 Options A version Display version and license information c aes128 aes192 aes256 none Encrypt in transit file data using the specified cipher This option overrides...

Страница 147: ...age 126 Note When filtering rules are found in aspera conf they are applied before rules given on the command line E and N exclude newer than mtime exclude older than mtime Exclude files but not direc...

Страница 148: ...as overwrite diff compare size resume k 2 Compare sparse checksum and resume if they match same as overwrite diff compare md5 sparse resume k 3 Compare full checksum and resume if they match same as o...

Страница 149: ...iles at the destination with source files of the same name based on the method Default always Use with compare and resume method can be the following always Always overwrite the file never Never overw...

Страница 150: ...lize the available bandwidth up to the maximum rate When congestion occurs bandwidth is shared fairly by transferring at an even rate This option requires maximum target and minimum transfer rates l a...

Страница 151: ...fix must specify the URI in the same manner as the source paths For example if a source path includes an embedded passphrase the prefix must also include the embedded passphrase otherwise it will not...

Страница 152: ...3 client does not set Z the datagram size is the discovered MTU and the server logs the message LOG Peer client doesn t support alternative datagram size Ascp 4 Transfers with Object Storage Files tha...

Страница 153: ...eads and eight read threads on the client and eight meta threads and 16 write threads on the server ascp4 L tmp logs R tmp logs l1g scan threads 2 read threads 8 write threads 16 meta threads 8 data 1...

Страница 154: ...tbatch 0 1 Enable packet batching in read write Default 1 maxsize N Set the maximum stream length Default unlimited maxtime N Set the maximum stream duration in seconds Default unlimited maxidle N Set...

Страница 155: ...mmended Rate Settings for Video Streams ascp4 Option Description Recommendation m Minimum rate Take the encoding rate of the transport stream and add 1 Mbps l Target rate Take the minimum rate and add...

Страница 156: ...one read threads 1 write threads 1 udp 233 3 3 3 3000 loopback 1 ttl 2 udp localhost 3000 Read a TCP stream from 192 168 10 10 port 2000 and send it to 10 10 0 51 On 10 10 0 51 write the stream to loc...

Страница 157: ...back 0 ascp4 L opt test local 03 R opt test remote 03 DD m 12m l 15m mode send host 10 132 117 2 user root read threads 1 write threads 1 compression none udp 233 33 3 3 3001 sndbufsz 100MB ifaddr 10...

Страница 158: ...ccess Run the following command to unset a docroot and set a file restriction asconfigurator x set_user_data user_name username absolute AS_NULL file_restriction restriction The restriction can be set...

Страница 159: ...uted sources On file systems that have file system notifications changes in source file systems new files and directories deleted items and renames are detected immediately eliminating the need to sca...

Страница 160: ...nfiguration 1 This is the simplest and most common configuration of Watch Folder services Use an account that has read permissions for all your files and follow the instructions in Creating a Push Wat...

Страница 161: ...t with rund watch watchd and watchfolderd opt aspera bin asuserdata a For more information on configuring see Watch Service Configuration on page 213 Watch Folder Service Configuration on page 174 Con...

Страница 162: ...asperawatchfolderd as described in Choosing User Accounts to Run Watch Folder Services on page 160 For more information see Starting Aspera Watch Services and Creating Watches on page 211 and Creating...

Страница 163: ...ble d109d1bd 7db7 409f bb16 ca6ff9abb5f4 asrun send code 0 null Enable a Service Enabling a stopped service starts the service This command can be used to restart a service that stops due to an error...

Страница 164: ...hat use init d service asperarund status Aspera Run Server asperarund RUNNING 2 Select or create a user account to run your services Watch Folder services must be run under a user with access to every...

Страница 165: ...Watch Service Configuration on page 213 and Watch Folder Service Configuration on page 174 Your system is now ready for Watch Folders To create a push Watch Folder see Creating a Push Watch Folder wit...

Страница 166: ...torage IBM Aspera Shares endpoints must have version Shares version 1 9 11 with the Watch Folder patch or a later version To create a push Watch Folder 1 Prepare your computer as described in Getting...

Страница 167: ...rce_directory target path target_directory location type REMOTE host hostname port port authentication type authentication_mode user username pass password keypath key_file watchd scan_period scan_per...

Страница 168: ...an access key ID and secret Sample JSON syntax for each authentication type is provided following this table NODE_BASIC user The username for authentication Required Depending on the type of authenti...

Страница 169: ...file is the path to the Watch Folder configuration file If you do not know the daemon retrieve a list of running daemons by running the following command opt aspera bin aswatchfolderadmin query daemon...

Страница 170: ...directory are re transferred Restrictions on all Watch Folders Only local to remote push and remote to local pull configurations are supported Remote to remote and local to local are not supported Gro...

Страница 171: ...send l The output is similar to the following in this example the user is root asrun send code 0 services id d109d1bd 7db7 409f bb16 ca6ff9abb5f4 configuration enabled true run_as pass user root type...

Страница 172: ...e Set type to REMOTE for the remote server type REMOTE is assumed if host is specified REMOTE host The host IP address DNS hostname or URL of the remote file system Required The host can be specified...

Страница 173: ...age NFS Solaris AIX and Isilon file system scans triggered by the scan period are used to detect file changes In this case set the scan period to frequently scan for changes On operating systems that...

Страница 174: ...folders daemon_name For example opt aspera bin aswatchfolderadmin query folders root aswatchfolderadmin query folders Found a single watchfolder b394d0ee 1cda 4f0d b785 efdc6496c585 7 Test your Watch...

Страница 175: ...les across all drops When this number is exceeded drops are purged until the file count is less than the specified number 9223372036854775807 watchfolderd_raw_options raw_options Enable the use of new...

Страница 176: ..._period 10s meta version 0 name aspera_watchfolder drop detection_strategy COOL_OFF_ONLY cool_off 5s post_processing source type TRANSFER_NONE archive_dir watchfolder_sessions UUID _ DATETIME filters...

Страница 177: ...te_blk_size datagram_size rexmsg_size cipher AES128 overwrite DIFF resume NONE preserve_uid false preserve_gid false preserve_time false preserve_creation_time false preserve_modification_time false p...

Страница 178: ...pe REMOTE host host port port authentication type SSH NODE_BASIC user username pass password keypath key_file fingerprint ssh_fingerprint target path path id watchfolder_id cool_off 30s snapshot_creat...

Страница 179: ...dpoints enter 443 If authentication type is SSH then default is the value for tcp_port in the transport section default 22 If authentication type is NODE_BASIC then default is 9092 authentication type...

Страница 180: ...nsferred in the same transfer session post processed together and reported as a unit Watch Folders uses asperawatchd to detect file system modifications and continuously creates snapshots to compute t...

Страница 181: ...and reported as a unit drop detection_strategy COOL_OFF_ONLY cool_off 5s Field Description Default detection_strategy The strategy that Watch Folders uses to create drops when new files are added to...

Страница 182: ...NSFER_ARCHIVE Files in the source directory are moved to a final archive after successful transfer This option is not supported for sources in object storage TRANSFER_DELETE Files in the source direct...

Страница 183: ...NCLUDE and EXCLUDE Note An include rule must be followed by at least one exclude rule otherwise all files are transferred because none are excluded To exclude all files that do not match the include r...

Страница 184: ...e syntax as in the filters object N A The transport object Use to configure authentication to the remote host transport host 198 51 100 22 user aspx2 pass XF324cd28 token fiewle535etn23TEIW234n5sEWTns...

Страница 185: ...rver on page 16 Configuring Transfer Server Authentication N A tags Specify custom metadata in JSON format The tags object is passed directly to the ascp session For more information on writing custom...

Страница 186: ...min_rate 0B target_rate 10M tcp_port 22 udp_port 33001 read_blk_size write_blk_size datagram_size rexmsg_size cipher AES128 overwrite DIFF resume NONE preserve_uid false preserve_gid false preserve_t...

Страница 187: ...ata in transit Aspera supports three sizes of AES cipher keys 128 192 and 256 bits and supports two encryption modes cipher feedback mode CFB and Galois counter mode GCM The GCM mode encrypts data fas...

Страница 188: ...and servers version 3 9 0 and newer NONE Do not encrypt data in transit Aspera strongly recommends against using this setting All client and server versions Client Server Cipher Negotiation The follow...

Страница 189: ...he modification time of the destination file to that of the source false preserve_access_time Set the access time of the destination to that of the source The destination file has the access time of t...

Страница 190: ...Folder If a file does not match the growing file filter it is transferred by Ascp Note Growing files are only supported for local sources push Watch Folders and must be authenticated by a transfer us...

Страница 191: ...gram size MTU for FASP The detected path MTU cipher The encryption cipher that is used to encrypt streamed data in transit either NONE and AES128 AES128 completion_timeout How long to wait before the...

Страница 192: ...anges in the source directory Lower scan periods detect changes faster but can result in greater resource consumption particularly for object storage Note The value for scan period cannot be empty oth...

Страница 193: ...Update a Watch Folder s Configuration To update a Watch Folder configuration retrieve the Watch Folder s configuration make the desired changes and then save the configuration as a JSON file You cann...

Страница 194: ...max_user_watches 524288 etc sysctl conf 2 Increase the maximum number of inotify instances which correspond to the number of allowed Watch Services instances Retrieve the current value by running the...

Страница 195: ...lder source is in object storage IBM Aspera Shares endpoints must have version Shares version 1 9 11 with the Watch Folder patch or a later version To create a push Watch Folder with the API 1 Prepare...

Страница 196: ...the next step 5 Confirm that the services are running For each service run the following command curl ki u node_username node_password X GET https localhost 9092 rund services service_id The state is...

Страница 197: ...s Zone IDs for example eth0 can be appended to the IPv6 address N A port The port to use for authentication to the remote file system By default if the authentication type is SSH then the SSH port for...

Страница 198: ...Windows macOS asperawatchd uses the file notifications as the primary means for detecting changes and the scan period serves as a backup In this case the default value of 30 minutes is usually accepta...

Страница 199: ...Growing files are only supported for local sources push Watch Folders and must be authenticated by a transfer user password or SSH key file The transfer user cannot be restricted to aspshell and the...

Страница 200: ..._user root admin impersonation 4 Create a Watch Service on the remote server This approach requires that you have node credentials for the remote server a Create a JSON configuration file for the remo...

Страница 201: ...rs and growing file handling A basic pull Watch Folder configuration has the following syntax source path source_directory location type REMOTE host ip_address port port authentication type authentica...

Страница 202: ...or authentication depending on the type of authentication N A target path The target directory on the local computer relative to the transfer user s docroot N A watchd identifier The daemon associated...

Страница 203: ...F324cd28 H X aspera WF version 2017_10_23 X POST d watchfolder_conf json https 198 51 100 22 9092 v3 watchfolders id b394d0ee 1cda 4f0d b785 efdc6496c585 8 Verify that the Watch Folder is running curl...

Страница 204: ...a configuration option that was not set Errors with ascp transfers are displayed similarly in the transport section curl ks user watchfolder_admin XF324cd28 H X aspera WF version 2017_10_23 X GET htt...

Страница 205: ...rl k user node_api_user node_api_password H X aspera WF version 2017_10_23 X GET https host node_api_port v3 watchfolders 2 Get the ID of the failed drop curl k user node_api_user node_api_password H...

Страница 206: ...aspera WF version 2017_10_23 is required when submitting POST PUT and GET requests to v3 watchfolders on servers that are version 3 8 0 or newer This enables Watch Folders to parse the JSON source an...

Страница 207: ...essfully deleted Configuring Custom Watch Folder Permissions Policies By default users are not allowed to perform any Watch Folders related actions unless they are configured with admin ACLs If you do...

Страница 208: ...te delete and view policies and assign users to policies These actions do not require that you specify a value for resources To allow all permissions use PERM_ PERM_CREATE_POLICY PERM_DELETE_POLICY PE...

Страница 209: ...IST_RESOURCES resources arn watchfolder wfd Assigning Node API Users to Policies Assign a user to one or more policies by running the following command curl k user node_api_user node_api_password X PU...

Страница 210: ...access to the source directory specified in the JSON configuration file You might have specified a destination that is not permitted by the docroot or restriction of the user running asperawatchfolder...

Страница 211: ...folder drive root file c Amazon S3 and IBM Cloud Object Storage S3 s3 Azure azu Azure Files azure files Azure Data Lake Storage adl Alibaba Cloud oss Google Cloud gs HDFS hdfs With a docroot or restri...

Страница 212: ...bject storage to which the user has access Users can create Watch Folders and Watch services on files or objects only within their docroot or restriction Note Users can have a docroot or restriction b...

Страница 213: ...hed by the Aspera Watch Service To create a watch users subscribe to a Watch Service and specify the path to watch run the following command where daemon is the username used to start the asperawatchd...

Страница 214: ...a conf setting Description Default watch_log_dir log_dir Log to the specified directory This setting applies to both the Watch Service and Watch Folders services The Aspera logging file Log Files on p...

Страница 215: ...eriod of an existing subscription Set the Default Scan Period When Upgrading from 3 7 4 or earlier to 3 8 0 or later To update the default scan period that is applied during the migration run the foll...

Страница 216: ...ing changes and the scan period serves as a backup In this case the default value of 30 minutes is usually acceptable and no change is necessary To never scan and rely entirely on file notifications s...

Страница 217: ...scription ID later 2 Create a snapshot opt aspera bin aswatchadmin create snapshot daemon subscription_id If you do not have the subscription ID run the following command opt aspera bin aswatchadmin q...

Страница 218: ...hat contain ASCII characters such as or are not deleted and an error is logged CAUTION asdelete follows symbolic links which can result in files being deleted that are not within the target directory...

Страница 219: ...irectional synchronization Aspera Sync runs with a bi directional option For a multi directional synchronization one session is run for each peer to remain sync Any topology that has an acyclic graph...

Страница 220: ...nce on page 234 This mode should be used for one_time operations or for periodic scheduled synchronizations where file systems do not support event based change notification For the latter async can b...

Страница 221: ...session one async process execution for each remote peer Any number of async processes can be run concurrently and any number of peers can be synchronized concurrently however a downstream peer cannot...

Страница 222: ...leted or the change occurs on both endpoints concurrently such that the newer version cannot be reliably determined Aspera Sync reports such conflicts and does not modify either file system leaving th...

Страница 223: ...ame async_db_dir db_dir This setting overrides the remote database directory specified by the client with the B option Note If the transfer user s docroot is a URL such as file then async_db_dir must...

Страница 224: ...c Value has the syntax sqlite lock_style storage_style Default undefined lock_style Specify how async interfaces with the operating system Values depend on operating system Unix based systems have the...

Страница 225: ...te_grant_mask Specify the mode for newly created directories if directory_create_mode is not specified If specified directory modes are set to their original modes plus the grant mask values This opti...

Страница 226: ...GUI shows transfers associated with a Aspera Sync job in which the remote user aspera is pushing files to the server folder for Project X You can configure the server and client reporting to the Aspe...

Страница 227: ...c link If a file with the same name exists at the destination the symbolic link does not replace the file Copy force Client only Copy only the symbolic link If a file with the same name exists at the...

Страница 228: ...s a database snap db that is stored on both the local client computer and the remote server computer The database records the state of the file system at the end of the last async session and the next...

Страница 229: ...sp ex2 snap db On the remote computer server opt aspera var private asp ex2 snap db storage users ex2 for transfer cache Changing Synchronization Direction Between Runs of the Same Session Changing di...

Страница 230: ...ze with AWS S3 storage see Synchronizing with AWS S3 Storage on page 253 1 Confirm that both endpoints have Aspera Sync enabled licenses and that the remote endpoint is running an Aspera transfer serv...

Страница 231: ..._TOKEN or in the command line using the W token_string or token token_string option For example use i and specify the path to Morgan s SSH private key in their home folder async L C Users Morgan Asper...

Страница 232: ...spera Sync is push or bidirectional use local mount signature If the remote endpoint is on a NFS or CIFS mount and the Aspera Sync is pull or bidirectional use remote mount signature 11 Specify the lo...

Страница 233: ...1 data R morgan async log B morgan async db K bidi t Note When synchronizing between Unix like operating systems you can also preserve the user IDs uid and group IDs gid from the source to the destina...

Страница 234: ...numeric characters plus _ and characters Note If your remote host is an Aspera cluster ensure that your session name is unique by naming the session with a descriptive string followed by the UUID of t...

Страница 235: ...dir are synchronized with newer versions of files and directories overwriting older versions in either ldir or rdir by default Using continuous mode C Continuous mode is supported only when the file s...

Страница 236: ...a directory s modification time has not changed compared to the Aspera Sync database async in non continuous mode skips scanning the directory This option makes scanning static directory structures fa...

Страница 237: ...but the server must support and permit it Cipher rules The encryption cipher that you are allowed to use depends on the server configuration and the version of the client and server When you request...

Страница 238: ...Server v3 9 0 AES XXX Server v3 8 1 or older AES XXX Client v3 9 0 AES XXX GCM GCM server refuses transfer GCM server refuses transfer Client v3 9 0 AES XXX CFB server refuses transfer CFB CFB CFB Cl...

Страница 239: ...nd the target files have matching inodes This option is supported only between Unix based platforms If dedup inode is used in a continuous sync Aspera recommends using the scan interval option copy Af...

Страница 240: ...ize Use the specified block size for writing size is an integer with units of K M or bytes Default 64 MB g size read block size size Set block size for reading size is an integer with units of K M or...

Страница 241: ...type type can be sha1 md5 sha1 sparse md5 sparse or none A value of none is equivalent to a size check only and async will not detect a change in timestamp Default sha1 sparse for local storage none f...

Страница 242: ...are added to the directory after the start of the async session but not existing files With no scan Aspera Sync relies entirely on file system notifications to detect changes As a result if a directo...

Страница 243: ...not synchronized when only the ACL is modified or when only the ACL and filename are modified ACLs are not preserved for directories On Windows the ACLs that are created for files that are transferred...

Страница 244: ...target directory that is inside your source directory remote force stat Force the remote Aspera Sync to retrieve file information even if no changes were detected by scanning or file system notificat...

Страница 245: ...iously found file that does not have multiple hardlinks it is considered a rename and the remote file is renamed accordingly Usage note This option can be used only on file systems with persistent ino...

Страница 246: ...d specially in the argument to r or remote dir W token_string token token_string Use the specified authorization token The token type sync push sync pull or sync bidi must match the direction push pul...

Страница 247: ...eads 4 R c logs 200 d c data r bobcat 192 168 4 24 C data K push l 500m Details Specifying the logging locations L and R is optional Adding 200 to the end of the log directory value allows the logs to...

Страница 248: ...kipped SYNCHRONIZED del file deleted SYNCHRONIZED ddp dedup duplicate files present SYNCHRONIZED exs file exists SYNCHRONIZED mov file has changed renamed moved or different attributes Include and Exc...

Страница 249: ...ows FAT or NTFS file systems and macOS HPFS a file system search for DEBUG returns files Debug and debug In contrast async filter rules use exact comparison To match both Debug and debug in a async fi...

Страница 250: ...are equivalent to exclude from G Specifying Rules in aspera conf Rules can be specified in aspera conf and applied to sessions run by a specific user or all users as they are for ascp sessions Rules i...

Страница 251: ...f abcefg abc abcde abc z abcdef abc d abc abc def adef cdef abcdef ade abc def zdef def 2def bdef def def abc def zdef def 2def cdef def def xxxxx lower def cdef ydef Adef 2def def Globbing Extensions...

Страница 252: ...ized 1 Include files under top level directories Raw and Jpg Exclude all others async include Raw include Jpg exclude exclude 2 Same as Example 1 except also include directories starting with at any l...

Страница 253: ...76 or 1MB Continuous transfer Bidirectional transfer Example Command async N asyncTwoWay d fio S r admin 192 168 200 218 d mnt fio S w v00d00 l 100M a fair g 1M G 1M C K BIDI Example Output SYNCHRONIZ...

Страница 254: ...count becomes an Aspera transfer user 4 Set database and log directories for async These directories must be located in mnt ephemeral data The mnt ephemeral directory is no cost ephemeral storage that...

Страница 255: ...Objects in Object Storage Files that are uploaded to metadata compatible storage S3 Google Cloud and Azure can have custom metadata written with them by using the tags or tags64 option The argument i...

Страница 256: ...your cluster and click the Access Keys tab Click New and fill in the required information for a description of the fields see the Aspera Transfer Cluster Manager Admin and Usage Guide AoCts See https...

Страница 257: ...tch Service enables fast detection and transfer of new and deleted items For more information on using watches with ascp see Transferring and Deleting Files with the Aspera Watch Service on page 216 T...

Страница 258: ...d for the user who runs the service For example if you started a Watch Service under root you should see the root daemon listed when you run the following command opt aspera bin aswatchadmin query dae...

Страница 259: ...awatchd is used only for pull requests by that user To configure the Watch Service database as the default run the following command asconfigurator x set_node_data async_watchd redis hostname 31415 do...

Страница 260: ..._address port domain This setting applies to both the Watch Service and Watch Folders services redis 127 0 0 1 31415 watchd_max_directories max_directories The maximum number of directories that can b...

Страница 261: ...file to determine whether or not to use asperawatchd for the session To pull files start a Aspera Sync session with the K pull option For example async N watch_pull d data D1 r adminuser 10 0 0 1 dat...

Страница 262: ...ta R11 K BIDI In this example the client on Host A starts the Aspera Sync session The asperawatchd service on Host B 10 54 44 194 scans the data D1 directory mounted by Host A and passes the snapshot...

Страница 263: ...ir path Specify the local Aspera Sync directory E number erase number Delete the specified file record by number F force Allow changes while database is in use f file info Report the status of all fil...

Страница 264: ...are older This option is only applied if async has been run using the exclude dirs older than option v verbose Increase the verbosity of summary s or file info f x init Delete all file system snapsho...

Страница 265: ...ync but is in error for the underlying ascp process For example when async is run with checksum none and access to the file is denied async does not open the file to calculate a checksum so it does no...

Страница 266: ...n You must run your Aspera Sync session to or from a computer with an operating system that supports continuous mode Continuous Aspera Sync Direction Supported Aspera Sync Client OS Supported Aspera S...

Страница 267: ...and you want to synchronize the following directory and files on both computers My_documents Document1 Document2 Document3 If Document2 is changed on both computer A and computer B then when you run t...

Страница 268: ...u to resolve the original conflict after synchronization Requires access to only one endpoint If you only have access to one endpoint want to preserve changes on both sides but do not want to resolve...

Страница 269: ...mand in the ssh folder The program prompts you for the key pair s filename Press ENTER to use the default name id_rsa For a passphrase you can either enter a password or press return twice to leave it...

Страница 270: ...ents of the directory media wmv Exclude files within the directory Exclude all other directories Preserve the owner and group ID Preserve access and modification time stamps on files No encryption Tra...

Страница 271: ...rom file I include from file Include filter text file with paths for inclusion See Include and Exclude Filtering Rules on page 248 exclude from file E exclude from file Exclude filter text file with p...

Страница 272: ...tures and functionality An HTTPS by default port 9092 and HTTP by default port 9091 interface An API that uses JSON data format The API is authenticated and the node daemon uses its own application le...

Страница 273: ...e file_restriction restriction Where username is the system user s username is a delimiter and restriction is specific to the storage type and path Storage Type Format Example local storage For Unix l...

Страница 274: ...into ssh and rename it authorized_keys or append the public key to authorized_keys if the file already exists cp opt aspera var aspera_tokenauth_id_rsa pub home aspera_user_1 ssh authorized_keys c Ens...

Страница 275: ...asperanoded or for Linux systems that use init d service asperanoded restart Node Admin Tool Use the asnodeadmin tool to manage add modify delete and list Node API users Root privileges are required S...

Страница 276: ...ope role for bearer create token key length Specify the RSA key length for bearer create user id user_id Specify the user id for bearer create bearer verify Verify bearer token f conf_filename Specify...

Страница 277: ...server section of aspera conf Asconfigurator Use the following syntax substituting option with the option from the following table and value with the desired value opt aspera bin asconfigurator x set_...

Страница 278: ...rt asperanoded transfers_retry_duration transfers_retry_duration If a transfer fails node will try to restart it for the specified time default 20m If a transfer restarts and makes some progress then...

Страница 279: ...415 Before changing this value you should back up your database See Backing up and Restoring the Node User Database Records on page 282 Restart asperanoded and the Redis database ssl_ciphers ssl_ciphe...

Страница 280: ...ery transfers that are associated with this access key through the events endpoint The server configuration can be overridden by the access key configuration This option must be enabled for event repo...

Страница 281: ...speranoded restart Reload the Node Configuration sudo opt aspera bin asnodeadmin reload Restart asperanoded and the Redis database 1 Stop asperanoded systemctl stop asperanoded or for Linux systems th...

Страница 282: ...e Redis database sudo opt aspera bin asnodeadmin r filepath database backup Note If you do not want to keep users that have been added since the last backup operation delete them after performing the...

Страница 283: ...de is using the default port for the Redis database port 31415 If your deployment uses a different port for Redis substitute it in the commands accordingly 1 Verify that the original node and new node...

Страница 284: ...lude pem crt cer and key and are Base 64 encoded ASCII files containing BEGIN CERTIFICATE and END CERTIFICATE statements Server certificates intermediate certificates and private keys can all be put i...

Страница 285: ...signed certificate Note Some certificate authorities provide a CSR generation tool on their website For additional information check with your CA 4 If required generate a self signed certificate You m...

Страница 286: ...icate or certificate bundle root certificate with chained or intermediary certificates from an authorized Certificate Authority For instructions on generating an SSL certificate see Setting up SSL for...

Страница 287: ...file pem cert_file cert file for asperanoded server Installing the SSL Certificates 1 Back up the default private key and self signed certificate using the following commands cd opt aspera etc cp aspe...

Страница 288: ...t examples Success The following sample output shows that verification was successful because verify return is 0 depth 2 C US O VeriSign Inc OU VeriSign Trust Network OU c 2006 VeriSign Inc For author...

Страница 289: ...thentication and Authorization Introduction to Aspera Authentication and Authorization HST Server can be configured to support SSH or HTTPS authentication and authorization for browsing and transfers...

Страница 290: ...ers or groups are configured to require token authorization only transfers initiated with a valid token transfer token basic token or bearer token are allowed to transfer to or from the server Token a...

Страница 291: ...ng purposes For more information on astokengen see Transfer Token Generation astokengen on page 293 Prerequisites In order to create transfer tokens with the Node API you must set up HST Server for th...

Страница 292: ...ths destination_root http serengeti com 9091 files upload_setup The response output is the following from which you extract the token string ATV7_HtfhDa JwWfc6RkTwhkDUqjHeLQePiOHjIS254_LJ14_7VTA HTTP...

Страница 293: ...users to generate and decode transfer tokens Unless you are creating a transfer token for an Ascp 4 session which requires that you use astokengen with the full paths option Aspera recommends using th...

Страница 294: ...download token Each pair of lines encodes one source and one destination and blank lines are ignored For example monday first_thing txt archive monday texts first_thing monday next_thing txt archive...

Страница 295: ...ause astokengen to fail Paired upload The destination is prepended to the destinations in the paired list file and they are encoded into the token The destinations are in the odd numbered lines of the...

Страница 296: ...triction configured in aspera conf rather than a docroot If a docroot is configured access key creation and use fails Access keys must specify the storage path Although they can be created with no sto...

Страница 297: ...eys d access_key_config json where access_key_config json is the access key configuration file For example curl ki u nodeadmin superP 55wOrD X POST https localhost 9092 access_keys d nodeadmin ak_clie...

Страница 298: ...transfer Optional JSON object The transfer configuration object Available as of 3 8 0 cipher Optional String The encryption mode and minimum cipher key length allowed by the server for transfers that...

Страница 299: ...g network or storage capacity if the client also requests a high minimum transfer rate that is not capped by the server This can decrease transfer performance and cause problems on the target storage...

Страница 300: ...aej_logging Optional Boolean Set to true to enable reporting to the IBM Aspera on Cloud Activity app The access key configuration overrides the server configuration This option must be enabled for ac...

Страница 301: ...id external_id assume_role_session_name session_name Where path includes the bucket and file path If server side encryption is set to AWS_KMS then server_side_encryption_aws_kms_key_id is required and...

Страница 302: ...uch as https company blob core windows net temp sv 2014 02 14 sr c sig yfew 79uXE 3D st 2015 07 29T07 3A00 3A00Z se 2018 08 06T07 3A00 3A00Z sp rwdl Azure Files storage type azure files path share pat...

Страница 303: ...40developer gserviceaccount com IBM Cloud Object Storage COS S3 storage type ibm s3 path bucket path endpoint s3 api us geo objectstorage service networklayer com credentials type key access_key_id k...

Страница 304: ...ess to a specific area of a storage and authenticates that user to the storage Basic tokens are less restrictive than transfer tokens They can be used to transfer with any Aspera server that supports...

Страница 305: ...l other Aspera servers too To create a bearer token with asnodeadmin run the following command as a user with admin root permissions If you do not specify an SSL key file or directory you are asked if...

Страница 306: ...n your HST Server installation Overview Deploying HST Server as a high availability cluster enables you to leverage the high speed transfer capabilities of Aspera with continuous availability and auto...

Страница 307: ...hysical virtual or container on which IBM Aspera High Speed Transfer Server is installed Glibc 2 5 or higher SSH Server If you are using OpenSSH version 5 2 or higher is recommended These instructions...

Страница 308: ...s create or edit an existing opt aspera etc redis conf with the following values slave of primary_ip_address 31415 bind private_ip_address port 31415 daemonize yes pidfile opt aspera var run redis 314...

Страница 309: ...era etc redis sentinel log Save your changes and close the file 5 On each node configure HAProxy a Open opt aspera etc haproxy haproxy cfg template http haproxy 1wt eu download 1 4 doc configuration t...

Страница 310: ...y the configuration opt aspera sbin haproxy f opt aspera etc haproxy haproxy cfg c e Configure iptables to ACCEPT the Redis IP addresses Your cluster is now configured You can now launch and test it S...

Страница 311: ...lly can be cumbersome and error prone because correct syntax and structure are strictly enforced The asconfigurator utility enables you to edit aspera conf through commands and parses validates and wr...

Страница 312: ...feature settings for use with the Node API For parameters and values see Server Configurations on page 325 set_http_server_data Sets data in the HTTP fallback server section For parameters and values...

Страница 313: ...x command parameter value fitness fitness_rule fitness_template Fitness Rule Example Description cookie cookie wilcard_template The parameter value is applied if the cookie passed from the application...

Страница 314: ...bling a Vlink with an ID of 101 and a capacity of 100Mb s asconfigurator x set_trunk_data id 101 trunk_on true trunk_capacity 100000 Allowing only encrypted transfers asconfigurator x set_node_data tr...

Страница 315: ...ase section c Outputs configurations set in the central server section t Outputs configurations set in the HTTP server section a Outputs configurations set in all sections except the user and group se...

Страница 316: ...incoming transfers Values String authorization_transfer_out_external_provider_url The URL of the external authorization provider for outgoing transfers Values String authorization_transfer_in_external...

Страница 317: ...sfer_out_bandwidth_flow_target_rate_lock A value of false allows users to adjust the transfer rate for outgoing transfers A value of true prevents users from adjusting the transfer rate for outgoing t...

Страница 318: ...s users to adjust the bandwidth policy for outgoing transfers A value of true prevents users from adjusting the bandwidth policy for outgoing transfers Values false default true transfer_in_bandwidth_...

Страница 319: ...he location path where file manifests are created Values Absolute path pre_calculate_job_size The policy of calculating total job size before a transfer If set to any the client configuration is follo...

Страница 320: ...by Aspera users Values Absolute path read_allowed Whether users are allowed to transfer files from the docroot in other words download from the docroot Values true default false write_allowed Whether...

Страница 321: ...mmand opt aspera bin asuserdata Vlink Configurations trunk_id The ID of the Vlink Values Number 1 255 trunk_on Whether the Vlink is enabled true or disabled false Values true false trunk_capacity The...

Страница 322: ...error Values ignore default exit compact_on_startup Whether to compact the local transfer history database on startup note that this may take awhile Values ignore default exit files_per_session The n...

Страница 323: ...lows the HTTP Fallback server to accept transfer requests on all network interfaces Values Network interface address default 0 0 0 0 restartable_transfers Whether interrupted transfers should resume a...

Страница 324: ...password The password for the database server Values String database_name The name of the database used to store Aspera transfer data Values String threads The number of parallel connections used for...

Страница 325: ...t 1 ignore_empty_files Whether to block the logging of zero byte files true or not false Values false default true ignore_skipped_files Whether to block the logging of skipped files true or not false...

Страница 326: ...er HTTP is enabled for asperanoded on the port configured for http_port true or not false Values false default true enable_https Whether HTTPS is enabled for asperanoded on the port configured for htt...

Страница 327: ...documentation for the default list of ciphers Values Colon delimited list ssl_protocol The minimum allowed SSL protocol Higher security protocols are always allowed tlsv1 default tlsv1 1 tlsv1 2 Aspe...

Страница 328: ...e first character is a separator preferably a which can be used to set multiple hosts For example 10 0 23 123 33001 10 0 23 124 33001 10 0 23 125 33001 Values Character separator IP address Character...

Страница 329: ...e following command opt aspera bin asuserdata Parameters and Values transport_cipher The encryption cipher to use for transfers Values aes 128 default aes 192 aes 256 none ssl_ciphers The list of SSL...

Страница 330: ...gnore the permission denied message after entering the password which is discussed in next steps 4 Applied authentication method is enabled in SSH If you can establish a SSH connection but it returns...

Страница 331: ...he first time running htpasswd to create the webpasswd file Do not use the c option otherwise If you still encounter connection problems after going through these steps contact Technical Support on pa...

Страница 332: ...e default of 10 MB For information on other logging configuration options see Server Logging Configuration for Ascp and Ascp 4 on page 78 Logging settings are configured by running asconfigurator comm...

Страница 333: ...ped or if you have modified the central_server or database sections in aspera conf then you need to restart the service Run the following command in a Terminal window to restart asperacentral systemct...

Страница 334: ...e that can include a substitutional string Supported strings name home The pathname can be in URI format special characters must be URL encoded A set of file system filters that use as a wildcard and...

Страница 335: ...curing the Aspera Application Securing Content in your Workflow Securing the Systems that Run Aspera Software The systems that run Aspera software can be secured by keeping them up to date by applying...

Страница 336: ...web UI you must also update the SshPort value in the WEB section of aspera conf For details see Configuring your Web UI Settings on page 25 Once this setting takes effect Aspera clients must set the...

Страница 337: ...mptyPasswords no e Disable root login CAUTION This step disables root access Make sure that you have at least one user account with sudo privileges before continuing otherwise you may not have access...

Страница 338: ...eract with the servers The instructions for Shares 1 9 x and Shares 2 x are slightly different see the section for your version HST Server 1 Restrict user permissions with aspshell By default all syst...

Страница 339: ...Files azure files Google Cloud Storage gs Hadoop HDFS hdfs The is a delimiter and you can add additional restrictions For example to restrict the system user xfer to s3 s3 amazonaws com bucket_xyz fol...

Страница 340: ...ryption If you require higher encryption change this value by running the following command asconfigurator x set_client_data transport_cipher value You can also specify the encryption level in the com...

Страница 341: ...self signed certificates Aspera recommends installing valid signed certificates These are required for some applications Securing Content in your Workflow 1 If your workflow allows enable server side...

Страница 342: ...tor x set_group_data group_name group_name transfer_encryption_content_protection_secret passphrase Important If the EAR password is lost or aspera conf is compromised you cannot access the data on th...

Страница 343: ...an be unencrypted To encrypt a file before moving it to a computer with network access run the following commands to set the encryption password and encrypt the file export ASPERA_SCP_FILEPASS passwor...

Страница 344: ...sequential or random Default is sequential optional When set to sequential file size is calculated as size N 1 increment Where N is the file index for the first file N is one When set to random file...

Страница 345: ...disk The I O throughput the disk bus architecture such as RAID IDE SCSI ATA and Fiber Channel Network I O The interface card the internal bus of the computer CPU Overall CPU performance affects the t...

Страница 346: ...an leave some blank For some fields there will be a default value If you enter the field will be left blank Country Name 2 letter code US Your_2_letter_ISO_country_code State or Province Name full nam...

Страница 347: ...key in any directory as long as the paths are updated in your configuration file For additional information see Enable SSL Apache Enable SSL Apache Install and enable an SSL certificate for your HST S...

Страница 348: ...ebian 7 or older Ubuntu 14 10 or older sudo service apache2 restart CentOS 6 RHEL 6 sudo service httpd restart 4 Test your SSL connection Go to https your server ip or name to test your SSL setup This...

Страница 349: ...e application window The five logging levels to select from are Off Error Warn Info and Debug The system default is Info Redirecting Aspera Logging to a Different Location On Linux systems the applica...

Страница 350: ...n the case of Red Hat or CentOS 6 X service rsyslog restart Your Aspera log messages now appear in var log aspera log instead of var log messages SLES Suse systems On SLES Suse systems the transfer lo...

Страница 351: ...slog as follows var log messages var log secure var log maillog var log spooler var log boot log var log cron var log aspera log sharedscripts postrotate bin kill HUP cat var run syslogd pid 2 dev nul...

Страница 352: ...utable yes Ascp and Aspera Sync update the destination file from mutable to immutable However if the source file is changed back to mutable immutable no the change cannot be applied to the destination...

Страница 353: ...ny All rights reserved Licensed Materials Property of IBM 5725 S58 Copyright IBM Corp 2007 2019 Used under license US Government Users Restricted Rights Use duplication or disclosure restricted by GSA...

Результаты поиска

Содержание Aspera HST

Отзывы:

Похожие инструкции для Aspera HST

Бренды по названию

Популярные бренды

IBM Aspera HST, Руководство администратора

Результаты поиска

Содержание Aspera HST

Отзывы:

Похожие инструкции для Aspera HST

Бренды по названию

Популярные бренды