start:fs_dm:archiving:preparation

Preparing to archive

We suggest to compress and annotate data prior to archiving with the TSM archive:

  • compressing saves transfer time
  • annotation eases the interpretation of retrieved data (if an archive needs to be pulled back).

A smaller directory can be compressed in the standard way:

# assuming gzip compression
$ tar -czf <archivename>.tar.gz <directoryname>

You may speed-up the compression, on a login-node using a parallel compression tool like pigz:

$ module load tools/pigz
$ tar cf - <directoryname> | pigz -p 4 > <archivename>.tar.gz

If the directory you are working on is too big, you can run an interactive job, too:

$ module load tools/pigz
# an interactive job might look like:
$ srun -A <your account> -p parallel -C broadwell -t <appropriate time> -N 1 -c40 --pty bash -i 
<some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname>
  • start/fs_dm/archiving/preparation.txt
  • Last modified: 2022/05/17 20:17
  • by meesters