This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
start:fs_dm:archiving:preparation [2021/04/15 11:28] – [Compressing Directories] meestersstart:fs_dm:archiving:preparation [2022/05/17 20:17] (current) – [Compressing Directories] meesters
Line 8: Line 8:
 ===== Compressing Directories ===== ===== Compressing Directories =====
-Smaller directory can be compressed in the standard way:+A smaller directory can be compressed in the standard way:
 <code bash> <code bash>
Line 15: Line 15:
 </code> </code>
-For directory with many huge files, this will take some patience yours and others on the login-node where you are working on. An alternative is to reserve a full node and use parallel compression (again assuming gzipjust using the parallel version of it):+You may speed-up the compression, on a login-node using a parallel compression tool like ''pigz'': 
 +<code bash> 
 +$ module load tools/pigz 
 +$ tar cf - <directoryname> | pigz -p 4 > <archivename>.tar.gz 
 +If the directory you are working on is too bigyou can run an interactive job, too:
 <code bash> <code bash>
Line 23: Line 29:
 <some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname> <some node>:$ tar -I pigz -cf <archivename>.tar.gz <directoryname>
 </code> </code>
  • start/fs_dm/archiving/preparation.txt
  • Last modified: 2022/05/17 20:17
  • by meesters