Data Transfer

Overview

ssh and scp are simple but powerful tools to transfer data. They are usually available on all major operating systems and can be used out of the box. 

rsync usually needs to be installed separately. 

The hostname to be used for the datatransfer is: datatransfer.psi.ch

In the following examples, please replace  <username>, <data-path>,  <source-path>, <destination-path>, <local-data-path>, etc. with the needed usernames and paths!

[ADVANCED TOPIC] Connection Multiplexing

Each ssh connection to datatransfer.psi.ch will require you to provide your username/password and to approve the login via the MFA application.

While using SSH multiplexing, one can efficiently re-use an established SSH connection, authenticated via  MFA, for the subsequent connections, without the need to authenticate (with MFA) again.

You can configure this on your SSH by adding following configuration in your local ~/.ssh/config  ssh configuration file:

$ cat .ssh/config
Host datatransfer.psi.ch
  ControlMaster auto
  ControlPath ~/.ssh/mux-%r@%h:%p
  ControlPersist 86400
  HostName datatransfer.psi.ch
  User <username>

Before transferring / querying data, you first have to initialise the "master" connection (authenticate with your password/MFA )

$ ssh datatransfer.psi.ch "ls /test" 

In case the connection was successfully established all following ssh commands will be run via the main  connection, without additional authentication. You can now use any of the commands listed above to query and transfer your data like:

$ ssh datatransfer.psi.ch "ls /test/testdir-1"

To check the state of main connection:

$ ssh datatransfer.psi.ch -O check

To terminate main connection:

$ ssh datatransfer.psi.ch -O exit

The main connection lifetime is determined by the ConrolPersist value in your ~/.ssh/config  file. Please note that very long lived connections (more than few days) will be terminated on our side!

Prerequisites

In addition to the prerequisites listed at the beginning following prerequisites apply: 

• SSH access to datatransfer.psi.ch from your machine/organisation.
  • Please talk to your network/security team to allow this access in case your organisation is blocking outgoing SSH traffic!

Commands

Find Data

Data/files can be queried / listed as follows:

$ ssh <username>@datatransfer.psi.ch "ls /<data-path>"

or a recursive listing (might be slow)

$ ssh <username>@datatransfer.psi.ch "ls -R /<data-path>"

Retrieve Data

Data can be downloaded to your machine as follows:

• scp

  scp -r <username>@datatransfer.psi.ch:/<data-path> <destination-path>

   

• rsync

  $ rsync -av <username>@datatransfer.psi.ch:/<data-path> <destination-path>

   

Upload Data 

Uploading data is only possible into certain directories, please check the table above if your storage is writeable !

Uploading data can be done via:

• scp

  scp -r <local-data-path> <username>@datatransfer.psi.ch:/<destination-path>

• rsync

  rsync -av <local-data-path> <username>@datatransfer.psi.ch:/<destination-path>

Sometimes it might be useful to tell the rsync command what the ownership of the transferred files should be. This is achieved by passing the --chown argument:

rsync -rtv --chown=<username>:<pgroup> <local-data-path> <username>@datatransfer.psi.ch:/<destination-path>

Note also that we had to change the -a option (archive, which tries to preserve the original ownerships) to -rt (recursive + preserve times).

Troubleshooting

• Public-key based authentication is not possible on datatransfer.psi.ch !
• In very urgent cases - in case you have problems connecting to datatransfer.psi.ch - try to resort to one of the backup servers datatransfer-01.psi.ch and/or datatransfer-02.psi.ch . However do NOT use these hostnames in normal cases as we might stop these serves at any time!