User Tools

Site Tools


filesystems

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
filesystems [2017/11/04 05:45]
meesters [FTP (Filezilla)]
— (current)
Line 1: Line 1:
-<WRAP center round todo 90%> 
-''​This page is outdated with respect to Mogon II filesystems. We hope to provide more information,​ soon.''​. ​ 
-</​WRAP>​ 
  
-====== Filesystems ====== 
- 
- 
-<WRAP alert> 
-//**Please note**// 
- 
-Encryption is CPU intensive. If you want to transfer with encryption (e.g. with ssh) you can transfer via the login nodes. In order to achieve higher speeds, you can transfer to and from the fileserver (see below). 
- 
-Therefore ssh access to the mogonfs fileserver is not possible. 
-</​WRAP>​ 
-===== Access from outside of Mogon ===== 
- 
-==== FTP (Filezilla) ==== 
- 
-You can access the home and project directories via FTP (even from outside the university network) by connecting to ''​mogonfs.zdv.uni-mainz.de''​ (with **mandatory TLS** encryption((More information on the ZDV Homepage: https://​www.zdv.uni-mainz.de/​gruppenlaufwerke/#​Gruppenlaufwerk_ber_FTP_-_von_ausserhalbvon_Nicht-Domnen-Rechnern))). 
- 
-When using Filezilla, please select **File**->​**Sitemanager** (**Datei**->​**Servermanager** in German) to reach the dialog below, which shows a sample configuration. 
- 
-{{ ::​mogonftp.png |Exemplary configuration of FileZilla}} 
- 
-==== FTP (Shell) ==== 
- 
-For security reasons sftp will not work anymore with mogonfs. You have to use lftp and ftps for this.  
- 
-<code bash> 
-$ lftp -u <​username>​ mogonfs.zdv.uni-mainz.de 
-Password: 
-lftp <​username>​@mogonfs.zdv.uni-mainz.de:​~>​ mput *dat 
-xxx bytes transferred ​                             
-Total n files transferred 
-lftp <​username>​@mogonfs.zdv.uni-mainz.de:​~>​ mget *dat 
-xxx bytes transferred ​                             
-Total n files transferred 
-lftp <​username>​@mogonfs.zdv.uni-mainz.de:​~>​ quit 
-</​code>​ 
- 
-The first command will transfer all files ending on ''​dat''​ from the local machine to ''​mogonfs'',​ while the second command will transfer them from ''​mogonfs''​ to the local machine. A comprehensive manual is provided in the [[http://​linux.die.net/​man/​1/​lftp|man page]]. 
- 
-Retrieving all files from mogonfs from the command line, can be accomplished with this one-liner: 
-<code bash> 
-$ lftp -u <​username>​ -e "mget *dat" mogonfs.zdv.uni-mainz.de 
-Password: ​ 
-xxx bytes transferred ​                             
-Total n files transferred ​                                   ​ 
-lftp <​username>​@mogonfs.zdv.uni-mainz.de:​~>​ quit 
-</​code>​ 
- 
-With knowledge of the directory this one-liner can get the data from within that directory: 
-<code bash> 
-$ lftp -u <​username>​ -e "mget /​gpfs/​fs1/​home/<​username>/​somedir/​*dat"​ mogonfs.zdv.uni-mainz.de 
-Password: ​ 
-xxx bytes transferred ​                                                           ​ 
-Total n files transferred 
-lftp <​username>​@mogonfs.zdv.uni-mainz.de:​~>​ quit 
-</​code>​ 
- 
-''​lftp''​ can be fed with a scriptfile as well, it can hence be customized with tons of arguments. 
- 
-**Remember:​** ''/​project''​ is on ''/​gpfs/​fs2''​ for most users. 
- 
-In order to mirror a directory on mogon, you might want to refrain from lengthy calls to ''​rsync''​ + ''​ssh''​. ''​lfpt''​ can do the job, too: 
- 
- 
-<code bash> 
-$ lftp -u <​username>​ -e "​mirror -c /​gpfs/​fs1/​home/<​username>/​projects/​ncap/​ <path on localhost>"​ mogonfs.zdv.uni-mainz.de 
-</​code>​ 
- 
-If ''​mirror -c -e''​ is given, local files, which are not present on mogon anymore, are deleted on the local host. 
- 
-Sometimes it is useful to add the ''​--parallel''​ option, e.g. to transfer 3 files in concurrently:​ 
-<code bash> 
-$ lftp --parallel=3 ... 
-</​code>​ 
- 
-And, all in all, you might find typing your password over and over a bit cumbersome. Setting bookmarks might be a solution: 
-<code bash> 
-$ lftp <​user>​@mogonfs.zdv.uni-mainz.de 
-Password: 
-lftp <​user>​@mogonfs.zdv.uni-mainz.de:​~>​ set bmk:​save-passwords true 
-lftp <​user>​@mogonfs.zdv.uni-mainz.de:​~>​ bookmark add mogonfs.zdv.uni-mainz.de 
-lftp <​user>​@mogonfs.zdv.uni-mainz.de:​~>​ bookmark list 
-mogonfs.zdv.uni-mainz.de ftp://<​user>:​XXXX@mogonfs.zdv.uni-mainz.de/​ 
-lftp <​user>​@mogonfs.zdv.uni-mainz.de:​~>​ exit 
-</​code>​ 
-This should then work: 
-<code bash> 
-$ lftp mogonfs.zdv.uni-mainz.de 
-</​code>​ 
- 
-==== Windows Share (SMB/CIFS) ==== 
- 
-You can access the home and project directories from a Windows machine (within the university network) under the following URL: ''​[[\\mogonfs.zdv.uni-mainz.de\]]''​ or by explicitly connecting to ''​\\mogonfs.zdv.uni-mainz.de\<​username>''​ or ''​\\mogonfs.zdv.uni-mainz.de\project''​. 
- 
-{{ :​mogonfs.png |Exemplary navigation bar of Windows Explorer}} 
- 
- 
-===== GPFS ===== 
- 
-==== Quota ==== 
- 
-The commands needed to view the current quota usage on the GPFS are conveniently packaged within a small script which is available at ''/​cluster/​bin/​mogonquota''​. 
- 
-===== Backup ===== 
- 
-All user home directories and project directories are regularily backed up completely, with the exception of directories named ''​scratch'',​ ''​temp''​ or ''​tmp''​. Also directories named ''​work''​ are not backed up for project directories. 
-You should use those scratch directories to store data that is reproducible and/or temporary (e.g. the build directory of some software). 
- 
-==== Restore from backup ==== 
- 
-<WRAP important>​ 
-The following instructions are experimental 
-</​WRAP>​ 
- 
-On ''​mogon1''/''​login01'',​ you can view the available backup versions of a file (it is important that you use the full filesystem path - e.g. by prepending ''/​gpfs/​fs1''​ to paths in your home directory; sometimes adding ''​-subdir=yes''​ might be helpful): 
-<code bash> 
-[...@login01 ~]$ dsmc q backup -inactive -se=mogonfsbackup /​gpfs/​fs1/​home/​.../​.../​... 
-IBM Tivoli Storage Manager 
-Command Line Backup-Archive Client Interface 
- 
-[...] 
- 
-           ​Size ​       Backup Date                Mgmt Class           A/I File 
-           ​---- ​       ----------- ​               ---------- ​          --- ---- 
-         ​3,​769 ​ B  10/05/2013 03:​37:​56 ​            ​DEFAULT ​             A  /​gpfs/​fs1/​home/​.../​.../​... 
-</​code>​ 
- 
-You can then restore files from backup using the command (the last parameter is optional and can be used not to overwrite the file if it sill exists): 
-<code bash> 
-[...@login01 ~]$ dsmc restore -pic -inactive -se=mogonfsbackup /​gpfs/​fs1/​home/​.../​.../​... [/​optional/​path/​where/​to/​restore] 
-</​code>​ 
- 
-You will be presented with a terminal GUI window where you can select a backup version by entering the corresponding number and pressing enter and afterwards pressing ''​o''​ and enter. The file will then be fetched from the backup (which might take a while). When asking to restore all files within a directory be sure to add a wild card asterisk at the end of the selected path: ''/​*''​. ​ 
filesystems.1509770729.txt.gz · Last modified: 2017/11/04 05:45 by meesters