Preparing to archive
We suggest to compress and annotate data prior to archiving with the TSM archive:
- compressing saves transfer time
- annotation eases the interpretation of retrieved data (if an archive needs to be pulled back).
Compressing Directories
A smaller directory can be compressed in the standard way:
# assuming gzip compression $ tar -czf <archivename>.tar.gz <directoryname>
You may speed-up the compression, on a login-node using a parallel compression tool like pigz
:
$ module load tools/pigz $ tar cf - <directoryname> | pigz -p 4 > <archivename>.tar.gz
If the directory you are working on is too big, you can run an interactive job, too:
$ module load tools/pigz # an interactive job might look like: $ srun -A <your account> -p parallel -C broadwell -t <appropriate time> -N 1 -c40 --pty bash -i <some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname>