This page describes a number of data transfer options:
- methods, respectively tools
- file server addresses for those tools
- brief examples
For details of the file system and practical considerations please refer to the relevant wiki page. For an even more practical approach and more information please consider following an introductory course.
Addresses to connect with ssh/scp
In order to contact our HPC systems with
ssh (used by
rsync) address; these are the same as described in our notes on login procedures:
miil03from within the university net for MOGON II.
mogon, too - provided you use the configuration as shown.
scp stands for secure copy and works with <code>ssh</code> (the secure shell). To use
scp for data transfer at this time, you must tunnel through the
HPCGATE. The basic command would be:
scp -oProxyJump=<username>@hpcgate.zdv.uni-mainz.de <TheFileToCopy> <username>@<service-node>:~/<PathToDestination>
However, when using an ssh-configuration as shown in HowTo-connect section, the command simplifies to:
scp <TheFileToCopy> mogon:<PathToDestination> # or in case of full directories: scp -r <TheDirectoryToCopy> mogon:<PathToDestination>
See the section below for detailed instructions.
When to use scp
|Use Cases||Abuse Cases||Note|
|transferring single / a few small files||transferring big data|
|need to transfer files securely|
scp you need to define the
source and the
destination. Here, a host needs to be given for either one or both,
For the purpose of the following code examples below we will refer to
server=<some name of a server in your institutes basement> mogon=<either mogon login access>
# to copy a <file> in the current location to a <remote> host into the home directory ('.'): scp <file> <remote>:. # if the username is different on the remote, you need to supply it: scp <file> <user>@<remote>:. # if you want the file to go to a different location on the remote, you need to supply it: scp <file> <remote>:/path/to/location/on/remote/. # you may rename the file, as with the cp program: scp <file> <remote>:<new_name> # an entire directory and subdirectory tree can be copied recursively with -r: scp -r <directory> <remote>:/path/to/location/on/remote/.
These examples should illustrate how copy data from a host to a remote. In order to retrieve data from a remote, the remote server has to take the place of the
source and, for instance, the
destination becomes your host, e. g.:
scp -r <remote>:/path/to/location/on/remote/ <path/on/your/machine>
You can invoke
scp on MOGON II as the host and treat your destination as the remote, too.
|Recursively copy entire directories. |
|Enables compression. Files are transferred in transferred mode - may or may not be faster|
“Direct copy” means to copy from the machine you are logged on to and from MOGON.
Here, you can invoke
scp for a single file:
scp <filename> <mogon>:/desired/path
or, using the
-r flag an entire directory:
scp -r <directory> <mogon>:/desired/path
Copy from remote host to MOGON and vice versa
The idea is to trigger a command on your desktop to transfer from a remote storage (e. g. an institutes server) directly to MOGON II without the need to transfer to your desktop and subsequently to MOGON II.
The command line will need an additional flag and the names of both servers:
scp $server:<path to copy from server> mogon:<destination path on mogon> # example scp $server:/data/scripts/script.sh mogon:./bin/.
In order to copy entire directories use the
-r flag, see above.
serveris not set up for a direct transfer (e.g. when the ssh-server is not set-up), the
-3flag can be used to transfer to the local host.
rsync is a utility to transfer and synchronize files between computers (or external drives).
When to use rsync
|Use Cases||Abuse Cases|
|need to synchronize directories||no need to synchronize directories|
- The need to synchronize often arises in software development, when attempting to develop on a remote system and then transferring to an HPC in order to compile. This is an artificially imposed need. Consider to develop an HPC system and to use version management (e.g. the ZDV gitlab server) for your development cycle. This means shorter turn-around times and better testing opportunities.
- Other needs might arise from project needs. In the case of big data which need to be in sync, please consider
rsync is straight forward and best shown by a simple example:
Assuming you want to synchronize a
<source> directory on your desktop with a known
destination (a path) on
<mogon> stands for the address to be used with <code>ssh</code>/<code>scp</code> as
rsync -avzh <source> <mogon>:<destination>
-ais for archive, meaning it will preserve all the permissions , links , dates etc.
-vis for “verbosely report what you are doing” (can frequently be omitted)
-zis used for compressing of data during the transfer (may or may not be faster, see scp-comment)
-hask to display all output in a human readable format.