start:fs_dm:filesystems:transfer

Transfer Options

This wiki page describes a number of data transfer options:
  • methods, respectively tools
  • file server addresses for those tools
  • brief examples

For details of the file system and practical considerations please refer to the relevant wiki page. For an even more practical approach and more information please consider following an introductory course.

In order to contact our HPC systems with ssh (used by scp and rsync) address; these are the same as described in our notes on login procedures:

  • miil01.zdv.uni-mainz.de, miil02.zdv.uni-mainz.de or miil03.zdv.uni-mainz.de, resp. miil01, miil02 or miil03 from within the university net for MOGON II. Note that you can simply use mogon, too - provided you use the configuration as shown.

scp stands for “secure copy” and works with ''ssh'' (the secure shell) To use scp for data transfer at this time, you must tunnel through the HPCGATE. The basic command would be:

scp -oProxyJump=<username>@hpcgate.zdv.uni-mainz.de <TheFileToCopy> <username>@<service-node>:~/<PathToDestination>

However, when using a ssh-configuration as shown in HowTo-connect section, the command simplifies to:

scp <TheFileToCopy> mogon:<PathToDestination>
# or in case of full directories:
scp -r <TheDirectoryToCopy> mogon:<PathToDestination>

See the section below for detailed instructions.

To use scp you need to define the source and the destination. Here, a host needs to be given for either one or both, source and destination.

Conventions

For the purpose of the following code examples below we will refer to

server=<some name of a server in your institutes basement>
mogon=<either mogon login access>

Example

# to copy a <file> in the current location to a <remote> host into the home directory ('.'):
scp <file> <remote>:.
 
# if the username is different on the remote, you need to supply it:
scp <file> <user>@<remote>:.
 
# if you want the file to go to a different location on the remote, you need to supply it:
scp <file> <remote>:/path/to/location/on/remote/.
 
# you may rename the file, as with the cp program:
scp <file> <remote>:<new_name>
 
# an entire directory and subdirectory tree can be copied recursively with -r:
scp -r <directory> <remote>:/path/to/location/on/remote/.

These examples should illustrate how copy data from a host to a remote. In order to retrieve data from a remote, the remote server has to take the place of the source and, for instance, the destination becomes your host, e. g.:

scp -r <remote>:/path/to/location/on/remote/  <path/on/your/machine>

You can invoke scp on MOGON II as the host and treat your destination as the remote, too.

FlagMeaning
-r Recursively copy entire directories. source and destination are considered directory trees. Also see cp.
-C Enables compression. Files are transferred in transferred mode - may or may not be faster

“Direct copy” means to copy from the machine you are logged on to and from MOGON.

Here, you can invoke scp for a single file:

scp <filename> <mogon>:/desired/path

or, using the -r flag an entire directory:

scp -r <directory> <mogon>:/desired/path

The idea is to trigger a command on your desktop to transfer from a remote storage (e. g. an institutes server) directly to MOGON II without the need to transfer to your desktop and subsequently to MOGON II.

The command line will need an additional flag and the names of both servers:

scp $server:<path to copy from server> mogon:<destination path on mogon>
# example
scp $server:/data/scripts/script.sh mogon:./bin/.

In order to copy entire directories use the -r flag, see above.

If the server is not set up for a direct transfer (e.g. when the ssh-server is not set-up), the -3 flag can be used to transfer to the local host.

rsync is a utility to transfer and synchronize files between computers (or external drives).

When to use rsync

Use Cases Abuse Cases
need to synchronize directories no need to synchronize directories

Note

  • The need to synchronize often arises in software development, when attempting to develop on a remote system and then transferring to an HPC in order to compile. This is an artificially imposed need. Consider to develop a HPC system and to use version management2) for your development cycle. This means shorter turn-around times and better testing opportunities.
  • Other needs might arise from project needs. In the case of big data which need to be in sync, please consider lftp.

Using rsync is straight forward and best shown by a simple example:

Assuming you want to synchronize a <source> directory on your desktop with a known destination (a path) on <mogon> (here, <mogon> stands for the address to be used with ''ssh''/''scp'' as rsync uses ssh.

rsync -avzh <source> <mogon>:<destination>

Here,

  • -a is for archive, meaning it will preserve all the permissions , links , dates etc.
  • -v is for “verbosely report what you are doing” (can frequently be omitted)
  • -z is used for compressing of data during the transfer (may or may not be faster, see scp-comment)
  • -h ask to display all output in a human readable format.

Efficiency Considerations

Why should you bother, rsync with scp works reasonably well after all, doesn't it?

For the following example we created files of 1 GiB size with dd if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock of=sample_${number}.txt. These files were transferred to MOGON I in one of two ways:

  • either by means of scp via one of the login-nodes on MOGON I. The call was scp <fname> mogon:<testfolder> to copy one file to MOGON 3). Copying a file to a local disk from Mogon I was done by changing source and destination to the scp command.
  • or by means of lftp. The command was lftp -u <username> -e “cd <testfolder>; put <fname>” mogonfs.zdv.uni-mainz.de. Likewise, copying a file onto a local disk from MOGON I was done by changing put to get.

To copy multiple files and asterisk (*) was used for scp and an asterisk together with mput/mget for lftp. Also, the testfolder was a project folder. Each transfer type was tested thrice.

For testing to transfer 100 GiB 100 files of 1 GiB were created and transferred.

As the plotted results show lftp outperforms scp by 25 - 30 %. While the actual variability may exceed the measured, lftp is consistently above 100 MB / s.

No compression was tested, as the data were random, however actual files (particularly text files) can profit a lot from switching on compressions upon transfer.

When scp, when lftp?

scp is more convenient. lftp is faster, due to omitting encryption. So, scp is fine, also in combination with rsync for small to medium sized files. Here, the difference is that for 100 files of 1 GiB scp took more than 24 min. lftp only little more than a quarter of an hour.

Get in touch with us, if your bandwith is substantially lower than the values reported here!


1)
in case of Mogon II still via login-nodes
3)
Here, we assume that ssh was configured as described in our wiki.
  • start/fs_dm/filesystems/transfer.txt
  • Last modified: 2022/07/01 11:31
  • by jrutte02