User Tools

Site Tools


Archiving on Mogon

While we try to ensure regular backups on Mogon, this does not include archiving solutions. These are kindly provided by our colleagues from the ZDV-UNIX group

Archiving Account

Before contemplating archiving solutions for your group, please check whether your group already possesses a ZDV-archive. If in doubt, please write a mail to with your inquiry.

Likewise you can apply for an archive at Please note, that external users will be send an invoice on a yearly basis. Please ask for conditions in your mail. Internal users should be aware that there is currently no charge to them.

Using an Archive

Instead of specifying the archive server always as, use the one assigned for your group. This will be tsm02 for most practical purposes, we therefore used that for our examples.

Retrieving Archive Information

The archiving software is wrapped with a script. In order to list filespaces which have been archived invoke it like

$ /cluster/bin/tsmarchive archivetest query filespace

Output will be like:

> IBM Tivoli Storage Manager
> Command Line Backup-Archive Client Interface
>   Client Version 7, Release 1, Level 6.5 
>   Client date/time: 08/18/2017 12:23:03
> (c) Copyright by IBM Corporation and other(s) 1990, 2017. All Rights Reserved. 
> Please enter your user id <ARCHIVETEST>:

Now, users need to confirm with their ID, which is identical to the node in question (this is information you obtain with your archiving account), and hitting Enter/Return.

Then, your password is required:

> Please enter password for user id "ARCHIVETEST": 

An alternative way is to provide the password with -password on the command line.

It is possible to inquire what has been archived, e.g. on /gpfs/fs1/:

$ /cluster/bin/tsmarchive archivetest query archive /gpfs/fs1/ -subdir=yes -password=<your password>

When knowing a given description, one can search for it:

$ /cluster/bin/tsmarchive archivetest query archive c:\ -su=y -descr="some description"

Archiving Stuff

Before contemplating to archive data, please separate data from other files (e.g. testing scripts which will fulfill no purpose in re-creating results or intermediate results, which do not merit archiving). Consider which files need archiving (results? or experimental raw data?). But most importantly: Limit the number of files to be archived by creating (compressed) tar balls or zipped directories. The reason is: Archiving hundreds of thousands of files is not a good idea - because of the technical overhead and the resulting loss in overview, too.

The following command would archive the bin/ directory. The associated description would merely be the time stamp: <cod bash> $ /cluster/bin/tsmarchive <archivename> archive bin/ </code>

A more meaningful description can be supplied on the command line, too:

$ /cluster/bin/tsmarchive archivetest archive
<directory or file to be archived> -subdir=yes -description=\"My super important sampling scripts for DESY beamline\" 

In order to archive directories recursively you need to specify the -subdir=yes flag and a trainling slash (/) has to be amended to the directory to be archived.

  • The archiving client will ask for the user id - it is the same as the archivename.
  • On the first time the archiving client will ask your for the password which has been provided. It will then state that this password has been expired - you will need to choose a new one.

To be useful descriptions should fulfill the following criteria to support preservation and re-use (as a rule of thumb):

  • The creating user and/or (sub) project (not necessarily == a mogon project or SLURM account).
  • Hierarchical relationship to other archived data or chronological relationship. (e.g. “raw data to experiment 1 of project X” or “10th sample of project X”).
  • If present: Relationship to other data storages (e.g. “after quality control of raw data from <number exp.> at ILL Grenoble”)

Try to be as concise as possible. Descriptions should be stored separately (e. g. a classic folder in your institute) tabulating stored data, dates and the description.

Known Issues

When invoking the archiving command the client might complain about existing files it cannot write to (particularly dsmerror.log and dsminstr.log1). Try to delete those and proceed.

Retrieving Data

$ /cluster/bin/tsmarchive archivetest retrieve <path you want to restore> -sub=yes -descr=\"<your description>\"

The description can be omitted, but in case of double entries (entries with identical paths but different descriptions) a description is mandatory to retrieve the correct archive version.

If a recursive retrieval for a given path is desired a trailing slash / is required to amend to your path, like when archiving.

Getting help

Getting help is as simple as:

$ /cluster/bin/tsmarchive archivetest help

or specific

$ /cluster/bin/tsmarchive archivetest help archive
This one will only cause a warning.
archiving/tsm.txt · Last modified: 2019/07/12 07:41 by jsteinka