Preparing to archive
We suggest to compress and annotate data prior to archiving with the TSM archive:
- compressing saves transfer time
- annotation eases the interpretation of retrieved data (if an archive needs to be pulled back).
Compressing Directories
Smaller directory can be compressed in the standard way:
# assuming gzip compression $ tar -czf <archivename>.tar.gz <directoryname>
For directory with many huge files, this will take some patience - yours and others on the login-node where you are working on. An alternative is to reserve a full node and use parallel compression (again assuming gzip, just using the parallel version of it):
$ module load tools/pigz # reservation on Mogon I $ srun -A <your account> -p nodeshort -t <appropriate time> -N 1 -c64 --pty bash -i <some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname> # or on Mogon II $ srun -A <your account> -p parallel -C broadwell -t <appropriate time> -N 1 -c40 --pty bash -i <some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname>