User Tools

Site Tools


filesystems:transfer

A Case for Efficient Transfer

Why should you bother, rsync with scp works reasonably well after all, doesn't it?

For the following example we created files of 1 GiB size with dd if=/dev/urandom of=1GB.bin bs=64M count=16 iflag=fullblock of=sample_${number}.txt. These files were transferred to Mogon I in one of two ways:

  • either by means of scp via one of the login-nodes on Mogon I. The call was scp <fname> mogon:<testfolder> to copy one file to Mogon1). Copying a file to a local disk from Mogon I was done by changing source and destination to the scp command.
  • or by means of lftp. The command was lftp -u <username> -e “cd <testfolder>; put <fname>” mogonfs.zdv.uni-mainz.de. Likewise, copying a file onto a local disk from Mogon I was done by changing put to get.

To copy multiple files and asterisk (*) was used for scp and an asterisk together with mput/mget for lftp. Also, the testfolder was a project folder. Each transfer type was tested thrice.

For testing to transfer 100 GiB 100 files of 1 GiB were created and transferred.

As the plotted results show lftp outperforms scp by 25 - 30 %. While the actual variability may exceed the measured, lftp is consistently above 100 MB / s.

No compression was tested, as the data were random, however actual files (particularly text files) can profit a lot from switching on compressions upon transfer.

When scp, when lftp?

scp is more convenient. lftp is faster, due to omitting encryption. So, scp is fine, also in combination with rsync for small to medium sized files. Here, the difference is that for 100 files of 1 GiB scp took more than 24 min. lftp only little more than a quarter of an hour.

Get in touch with us, if your bandwith is substantially lower than the values reported here!

FTP (Filezilla)

You can access the home and project directories via FTP (even from outside the university network) by connecting to mogonfs.zdv.uni-mainz.de (with mandatory TLS encryption2)).

When using Filezilla, please select FileSitemanager (DateiServermanager in German) to reach the dialog below, which shows a sample configuration.

Exemplary configuration of FileZilla

FTP (Shell)

For security reasons sftp will not work anymore with mogonfs. You have to use lftp and ftps for this.

$ lftp -u <username> mogonfs.zdv.uni-mainz.de
Password:
lftp <username>@mogonfs.zdv.uni-mainz.de:~> mput *dat
xxx bytes transferred                             
Total n files transferred
lftp <username>@mogonfs.zdv.uni-mainz.de:~> mget *dat
xxx bytes transferred                             
Total n files transferred
lftp <username>@mogonfs.zdv.uni-mainz.de:~> quit

The first command will transfer all files ending on dat from the local machine to mogonfs, while the second command will transfer them from mogonfs to the local machine. You can specify a base directory or URL where files should be placed with -O, e.g. mput -O /path/to/dir *dat. A comprehensive manual is provided in the man page.

Retrieving all files from mogonfs from the command line, can be accomplished with this one-liner:

$ lftp -u <username> -e "mget *dat" mogonfs.zdv.uni-mainz.de
Password: 
xxx bytes transferred                             
Total n files transferred                                    
lftp <username>@mogonfs.zdv.uni-mainz.de:~> quit

With knowledge of the directory this one-liner can get the data from within that directory:

$ lftp -u <username> -e "mget /gpfs/fs1/home/<username>/somedir/*dat" mogonfs.zdv.uni-mainz.de
Password: 
xxx bytes transferred                                                            
Total n files transferred
lftp <username>@mogonfs.zdv.uni-mainz.de:~> quit

lftp can be fed with a scriptfile as well, it can hence be customized with tons of arguments.

Remember: /project is on /gpfs/fs2 for most users.

In order to mirror a directory on mogon, you might want to refrain from lengthy calls to rsync + ssh. lfpt can do the job, too:

$ lftp -u <username> -e "mirror -c /gpfs/fs1/home/<username>/projects/ncap/ <path on localhost>" mogonfs.zdv.uni-mainz.de

If mirror -c -e is given, local files, which are not present on mogon anymore, are deleted on the local host.

Sometimes it is useful to add the –parallel option, e.g. to transfer 3 files in concurrently (available whit the mirror command):

$ lftp --parallel=3 ...

And, all in all, you might find typing your password over and over a bit cumbersome. Setting bookmarks might be a solution:

$ lftp <user>@mogonfs.zdv.uni-mainz.de
Password:
lftp <user>@mogonfs.zdv.uni-mainz.de:~> set bmk:save-passwords true
lftp <user>@mogonfs.zdv.uni-mainz.de:~> bookmark add mogonfs.zdv.uni-mainz.de
lftp <user>@mogonfs.zdv.uni-mainz.de:~> bookmark list
mogonfs.zdv.uni-mainz.de ftp://<user>:XXXX@mogonfs.zdv.uni-mainz.de/
lftp <user>@mogonfs.zdv.uni-mainz.de:~> exit

This should then work:

$ lftp mogonfs.zdv.uni-mainz.de

Windows Share (SMB/CIFS)

You can access the home and project directories from a Windows machine (within the university network) under the following URL: \\mogonfs.zdv.uni-mainz.de\ or by explicitly connecting to \\mogonfs.zdv.uni-mainz.de\<username> or \\mogonfs.zdv.uni-mainz.de\project.

Exemplary navigation bar of Windows Explorer

FTP addresses for Mogon I and II

The addresses for the two clusters are:

  • mogonfs.zdv.uni-mainz.de (Mogon I ) and
  • mogon2ftp.zdv.uni-mainz.de (Mogon II)
filesystems/transfer.txt · Last modified: 2019/03/12 13:41 by jrutte02