File Transfer
Description
The File Transfer service utilises the SFTP, SMB and NFS protocols to download and upload files to and from the respective target server in each domain. The available file transfer protocols can be combined freely. For example; the service can be configured to download files in the source domain using SFTP, and then uploading them in the destination domain using SMB.
The File Transfer service on the Upstream DD Engine continuously monitors the configured server in the source domain for new files. Once the service detects a file and deems it to be suitable for transfer, it downloads the file and transfers it to the File Transfer service on the Downstream DD Engine. The service on the Downstream DD Engine receives files and uploads them to the configured server in the destination domain.
As mentioned above, the Upstream service continuously monitors the remote file server. A file is deemed ready for transfer once the service has encountered it, unmodified, a number of consecutive times.
Transfer mode
The File Transfer service supports two different transfer modes, Dropzone and Mirror.
In Dropzone mode, once the Upstream service has transferred a file Downstream, it will remove that file from the server in the source domain. However, if the service detects that the file has been modified on the remote server while the service was transferring it, it will NOT be removed, but instead re-transferred.
In Mirror mode, once the Upstream service has transferred a file Downstream, it will leave it untouched on the server in the source domain. If the service detects that a file which has already been transferred is updated on the server in the source domain, it will trigger the service to re-transfer that file. Note that if a file is removed from the monitored directory on the server in the source domain, this removal will NOT be mirrored to the server in the destination domain.
Supported versions
The following versions of respective protocols are supported by the File Transfer service:
| Protocol | Supported version(s) |
|---|---|
| SFTP | 3 |
| SMB | 2 and 3 |
| NFS | 3 |
When the File Transfer service is configured to use the SMB protocol, it will negotiate with the remote server and use the highest mutually supported version.
Configuration
In this section, the File Transfer service parameters are described in detail. Parameters are divided into sections, File Transfer Client Upstream and File Transfer Client Downstream, in the same way as they are displayed when configuring the service using the DD Manager.
In DD Manager, as well as in the parameter descriptions below, mandatory parameters are indicated by asterisks (*).
File Transfer Client Upstream
The File Transfer Client Upstream service monitors a directory on a file server on the source network for files to transfer. When files are deemed stable, they are downloaded and transferred to the File Transfer Client Downstream service, before being removed from the file server.
| Parameter | Description | Format |
|---|---|---|
| Mode * | The transfer mode of the service. | Available options: Dropzone and Mirror |
| Protocol * | The file transfer protocol to use when communicating with the remote file server. | Available options: SFTP, SMB and NFS |
SFTP parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname (if DNS has been configured in Local admin) | Internet host name, see RFC 1123, section 2.1 |
| Port * | TCP port that the service will use to connect to the remote host. | Integer within range 1-65535 |
| Directory * | An absolute path to the directory where files will be either downloaded from or uploaded to. | String |
| Host key fingerprint | The SHA256 fingerprint of the remote server's public SSH host key. This is used to verify that the service is connecting to the correct server. This parameter is optional, but it is strongly recommended to provide the fingerprint of the server to prevent man-in-the-middle attacks. See Retrieving an SSH host key fingerprint for more information on how to retrieve a server's host key fingerprint. | String representation of the SHA256 fingerprint, with or without the SHA256: prefix |
| Keep-alive interval | The interval at which the service will send keep-alive packets to the server. This parameter is optional, leaving this field empty disables the sending of keep-alive packets. | Integer within range 1-300 (seconds) |
Authentication
| Parameter | Description | Format |
|---|---|---|
| Type * | The method to use when authenticating the configured user towards the remote server. | Available options: None, Password, Key |
| Username * | The username to use when authenticating towards the remote SFTP server. | String |
| Password * | The password to use when authenticating towards the remote SFTP server. This field is only available when the authentication type is set to Password. | String |
| Private key * | The private key file (PEM encoded) to use when authenticating towards the remote SFTP server. This field is only available when the authentication type is set to Key. See SSH public key authentication for more information. | String |
| Private key password * | The password to use when decrypting the provided private key. If the provided private key is NOT encrypted this field shall be left empty. This field is only available when the authentication type is set to Key. | String |
SMB parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname. | Internet host name, see RFC 1123, section 2.1 |
| Port * | TCP port that the service will use to connect to the remote host. | Integer within range 1-65535 |
| Share * | The name of the share to which the service shall connect. | String with alphanumerical characters, - and _ are allowed. Must start and end with an alphanumerical character |
| Domain * | The domain to use when authenticating towards the remote SMB server. | String |
Authentication
| Parameter | Description | Format |
|---|---|---|
| Username * | The username to use when authenticating towards the remote SMB server. | String |
| Password * | The password to use when authenticating towards the remote SMB server. | String |
NFS parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname. | Internet host name, see RFC 1123, section 2.1 |
| Mount point * | An absolute path to the mount point exported by the remote NFS server. | String |
| Version * | Version of the NFS protocol to use. | Available options: NFSv3 |
| User ID * | The numerical user ID to be used when interacting with the remote NFS server. | Integer within range 0-4294967295 |
| Group ID * | The numerical group ID to be used when interacting with the remote NFS server. | Integer within range 0-4294967295 |
Advanced
| Parameter | Description | Format |
|---|---|---|
| Poll interval * | The interval at which the service will poll the remote server for file updates (added, removed, modified). | Integer within range 1-3600 (seconds) |
| Stable threshold * | The number of consecutive polls a file must be unmodified before being deemed stable enough for download and transfer to the Downstream service. A value of 1 means the file is downloaded immediately at the first poll. | Integer within range 1-254 |
| Download concurrency * | The maximum number of concurrent sessions the service will establish towards the remote server and use for downloading files. Use this parameter with caution, too many concurrent transfers might cause the Downstream service to be overloaded, resulting in data loss. | Integer within range 1-255 |
File Transfer Client Downstream
The File Transfer Client Downstream service receives files from the File Transfer Client Upstream service and uploads them to a configured directory on a file server on the destination network.
| Parameter | Description | Format |
|---|---|---|
| Protocol * | The file transfer protocol to use when communicating with the remote file server. | Available options: SFTP, SMB and NFS |
SFTP parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname (if DNS has been configured in Local admin) | Internet host name, see RFC 1123, section 2.1 |
| Port * | TCP port that the service will use to connect to the remote host. | Integer within range 1-65535 |
| Directory * | An absolute path to the directory where files will be either downloaded from or uploaded to. | String |
| Host key fingerprint | The SHA256 fingerprint of the remote server's public SSH host key. This is used to verify that the service is connecting to the correct server. This parameter is optional, but it is strongly recommended to provide the fingerprint of the server to prevent man-in-the-middle attacks. See Retrieving an SSH host key fingerprint for more information on how to retrieve a server's host key fingerprint. | String representation of the SHA256 fingerprint, with or without the SHA256: prefix |
| Keep-alive interval | The interval at which the service will send keep-alive packets to the server. This parameter is optional, leaving this field empty disables the sending of keep-alive packets. | Integer within range 1-300 (seconds) |
Authentication
| Parameter | Description | Format |
|---|---|---|
| Type * | The method to use when authenticating the configured user towards the remote server. | Available options: None, Password, Key |
| Username * | The username to use when authenticating towards the remote SFTP server. | String |
| Password * | The password to use when authenticating towards the remote SFTP server. This field is only available when the authentication type is set to Password. | String |
| Private key * | The private key file (PEM encoded) to use when authenticating towards the remote SFTP server. This field is only available when the authentication type is set to Key. See SSH public key authentication for more information. | String |
| Private key password * | The password to use when decrypting the provided private key. If the provided private key is NOT encrypted this field shall be left empty. This field is only available when the authentication type is set to Key. | String |
SMB parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname. | Internet host name, see RFC 1123, section 2.1 |
| Port * | TCP port that the service will use to connect to the remote host. | Integer within range 1-65535 |
| Share * | The name of the share to which the service shall connect. | String with alphanumerical characters, - and _ are allowed. Must start and end with an alphanumerical character |
| Domain * | The domain to use when authenticating towards the remote SMB server. | String |
Authentication
| Parameter | Description | Format |
|---|---|---|
| Username * | The username to use when authenticating towards the remote SMB server. | String |
| Password * | The password to use when authenticating towards the remote SMB server. | String |
NFS parameters
Connection
| Parameter | Description | Format |
|---|---|---|
| Address * | Address of the remote host, either an IP address or a hostname. | Internet host name, see RFC 1123, section 2.1 |
| Mount point * | An absolute path to the mount point exported by the remote NFS server. | String |
| Version * | Version of the NFS protocol to use. | Available options: NFSv3 |
| User ID * | The numerical user ID to be used when interacting with the remote NFS server. | Integer within range 0-4294967295 |
| Group ID * | The numerical group ID to be used when interacting with the remote NFS server. | Integer within range 0-4294967295 |
Advanced
| Parameter | Description | Format |
|---|---|---|
| Upload concurrency * | The maximum number of concurrent sessions the service will establish towards the remote server and use for uploading files. | Integer within range 1-255 |
File constraints
Type
Only regular files are currently supported. Symbolic links (and any other non-regular file types) will be silently ignored.
When using the SMB protocol, the behavior when handling symbolic links depends on the server implementation/configuration. In some cases, symbolic links may be presented as regular files, while in other cases they may not be detected at all by the service.
Size
To prevent loss of data, the Downstream service temporarily stores incoming files on disk until it can verify that they have been successfully uploaded to the remote server. Therefore, the upper size limit of a file is determined by the amount of disk space available to the Downstream service.
Known limitations
The limitations listed below will be resolved in future versions of the service.
- If the File Transfer service fails to connect to its server, it will result in many reconnection requests.
- In some rare cases when using NFS in File Transfer, the service can freeze for several minutes when simultaneously transferring multiple large files (~10GB each). The file transfer will eventually be carried through, given a few minutes.
Retrieving an SSH host key fingerprint
The host key fingerprint of a server can be retrieved using the ssh-keygen and ssh-keyscan utility programs. The examples below are from a Linux environment, but the mentioned tools are available for other operating systems as well.
In the examples below SHA256:ipXfG2vYVrCEnY/E+1kcpMRHDmFKyCYpu9vg9TqJ52 is the fingerprint string which shall be entered into the service configuration.
From a remote server
In order to retrieve the host key fingerprint from a remote server, execute the following command in a terminal:
ssh-keyscan <server address> | ssh-keygen -E sha256 -lf -
For example:
ssh-keyscan 192.168.1.100 | ssh-keygen -E sha256 -lf -
Will yield the following result:
256 SHA256:ipXfG2vYVrCEnY/E+1kcpMRHDmFKyCYpu9vg9TqJ52E 192.168.1.100 (ED25519)
From a public key file
In order to retrieve the host key fingerprint given access to the public key file, execute the following command in a terminal:
ssh-keygen -E sha256 -lf <path to public key file>
For example:
ssh-keygen -E sha256 -lf /etc/ssh/ssh_host_key_ed25519_key.pub
Will yield the following result:
256 SHA256:ipXfG2vYVrCEnY/E+1kcpMRHDmFKyCYpu9vg9TqJ52E user@machine (ED25519)
SSH public key authentication
SSH key authentication leverages asymmetric encryption to provide authentication with higher security than traditional passwords. SSH key authentication also enables users to implement single sign-on across multiple SSH servers, as well as passwordless login in automated environments.
An asymmetric key pair consists of a public key and a private key. A user can upload its public key to an SSH server, and if the server considers the key trustworthy that user can in future authenticate itself using its private key.
The private key must remain only with the user, as possession of this key is proof of the users identity.
Generating a new key pair
An SSH key pair can be generated using the ssh-keygen utility program.
It is considered best practice to use the Ed25519 key algorithm, to generate such a key pair execute the following command:
ssh-keygen -t ed25519
If for some reason it is not possible to use the Ed25519 algorithm, you're recommended to use the RSA algorithm with a key length of at least 4096 bits. To generate such a key pair, execute the following command:
ssh-keygen -t rsa -b 4096
Key requirements
The configured private key must be PEM base64-encoded, and in one of the following formats:
- OpenSSH
- PKCS#5
- PKCS#8
The supported key algorithms are the following:
ssh-rsassh-ed25519rsa-sha2-256rsa-sha2-512ecdsa-sha2-nistp256ecdsa-sha2-nistp384ecdsa-sha2-nistp521